How to run the cookiecutter demo with fixed dependencies?

SvenLieber commented 1 year ago

Hi,

as part of reviewing this software for the Journal of Open Source Software (https://github.com/openjournals/joss-reviews/issues/5040) I tried to run a demo of the tool, but got some errors which I think might be related to https://github.com/OnroerendErfgoed/atramhasis/issues/749

My question is what the recommended procedure is to get the demo running with fixed dependecies. The only dependencies I see are

requirements-dev.txt

# Runtime requirements
--requirement requirements.txt

pyramid-debugtoolbar

requirements.txt

atramhasis

Description

I expected a running demo server, but instead got errors during the installation within the created python virtual environment about dependencies and eventually an openapi spec validator-related error when trying to run the server or create an RDF dump. See below for a detailed description.

Steps to reproduce

I followed the latest documentation on running a demo site with Cookiecutter.

Step 1: Install cookiecutter

$ mkvirtualenv atramhasis_demo
$ pip install -U cookiecutter

First I got an error because mkvirtualenv was not installed. Installing it properly turned out to be tricky, so instead I did the following:

$ python3 -m venv py-atramhasis-env
$ . py-atramhasis-env/bin/activate
$ pip install -U cookiecutter

Step 2: generate the demo site with cookiecutter worked without problems

$ cookiecutter gh:OnroerendErfgoed/atramhasis_demo_cookiecutter

Step 3: install dependencies

$ cd atramhasis_demo
$ pip install -r requirements-dev.txt
$ pip install -e .

This results in the following errors

ERROR: openapi-core 0.16.4 has requirement openapi-schema-validator<0.4.0,>=0.3.0, but you'll have openapi-schema-validator 0.2.3 which is incompatible.
ERROR: openapi-core 0.16.4 has requirement openapi-spec-validator<0.6.0,>=0.5.0, but you'll have openapi-spec-validator 0.4.0 which is incompatible.

Step 4: Setup the database worked without problems

$ alembic upgrade head
# fill the database with data
$ initialize_atramhasis_db development.ini

Step 5: Optionally create RDF dump

$ dump_rdf development.ini

This results in the following error:

from openapi_spec_validator.validation import openapi_spec_validator_proxy
ModuleNotFoundError: No module named 'openapi_spec_validator.validation'

Step 6: Install front-end dependencies worked without problems

$ cd atramhasis_demo/static
$ npm install

Step 7: Start the server

$ cd ../..
# start server
$ pserve development.ini

results in the following error

from openapi_spec_validator.validation import openapi_spec_validator_proxy
ModuleNotFoundError: No module named 'openapi_spec_validator.validation'

Environment

I work in an Ubuntu 20.04 virtual machine with Python 3.8.10 and pip 20.0.2 (in the virtual environment successfully created in the first step).

koenedaele commented 1 year ago

Looks like an issue with Python 3.8. I just ran your steps with Python 3.10 on Ubuntu 22.04 and that works, so I assume it's the Python 3.8 that makes the difference. The openapi libraries have been a hassle. We'll have a look at how we can fix this quickly.

As to the virtualenv, that's the tool I like to use, but there are indeed several different ways of installing a venv. I'll clarify that in the docs.

koenedaele commented 1 year ago

I'm not sure where openapi-core 0.16.4 is coming from. Normally it should be 0.13.8 which is required by pyramid_openapi3 0.14.3.

When executing these commands you normally get output on what's being installed. Could you post that?

$ cd atramhasis_demo
$ pip install -r requirements-dev.txt
$ pip install -e .

Or in you current virtualenvironment do a pip list

SvenLieber commented 1 year ago

Thanks, after installing Python 3.10 and using a virtual environment created from it, running the demo worked!

For completeness, here is the output of pip list from the Python 3.8 environment:

Package                   Version   Location                                            
------------------------- --------- ----------------------------------------------------
alembic                   1.9.2     
arrow                     1.2.3     
atramhasis                1.3.0     
atramhasis-demo           0.0       /home/slieber/repos/skos-editor-test/atramhasis_demo
attrs                     22.2.0    
Babel                     2.11.0    
backports.cached-property 1.0.2     
binaryornot               0.4.4     
bleach                    5.0.1     
CacheControl              0.12.11   
cachetools                5.2.1     
certifi                   2022.12.7 
chardet                   5.1.0     
charset-normalizer        3.0.1     
click                     8.1.3     
colander                  2.0       
cookiecutter              2.1.1     
decorator                 5.1.1     
dogpile.cache             1.1.8     
frozendict                2.3.4     
greenlet                  2.0.1     
html5lib                  1.1       
hupper                    1.11      
idna                      3.4       
importlib-metadata        6.0.0     
importlib-resources       5.10.2    
iso8601                   1.1.0     
isodate                   0.6.1     
Jinja2                    2.11.3    
jinja2-time               0.2.0     
jsonschema                4.17.3    
jsonschema-spec           0.1.2     
language-tags             1.2.0     
lxml                      4.9.2     
Mako                      1.2.4     
MarkupSafe                2.0.1     
more-itertools            9.0.0     
msgpack                   1.0.4     
openapi-core              0.16.4    
openapi-schema-validator  0.2.3     
openapi-spec-validator    0.4.0     
parse                     1.19.0    
PasteDeploy               3.0.1     
pathable                  0.4.3     
pbr                       5.11.1    
pip                       20.0.2    
pkg-resources             0.0.0     
pkgutil-resolve-name      1.3.10    
plaster                   1.1.2     
plaster-pastedeploy       1.0.1     
Pygments                  2.14.0    
PyLD                      2.0.3     
pyparsing                 3.0.9     
pyramid                   2.0       
pyramid-debugtoolbar      4.10      
pyramid-jinja2            2.10      
pyramid-mako              1.1.0     
pyramid-openapi3          0.15.0    
pyramid-rewrite           0.2       
pyramid-skosprovider      1.1.0     
pyramid-tm                2.5       
pyrsistent                0.19.3    
python-dateutil           2.8.2     
python-slugify            7.0.0     
pytz                      2022.7.1  
PyYAML                    6.0       
rdflib                    6.2.0     
requests                  2.28.2    
rfc3987                   1.3.8     
setuptools                44.0.0    
six                       1.16.0    
skosprovider              1.2.0     
skosprovider-getty        1.1.0     
skosprovider-rdf          1.3.0     
skosprovider-sqlalchemy   1.0.0     
SQLAlchemy                1.4.46    
stevedore                 4.1.1     
text-unidecode            1.3       
transaction               3.0.1     
translationstring         1.4       
typing-extensions         4.4.0     
urllib3                   1.26.14   
venusian                  3.0.0     
waitress                  2.1.2     
webencodings              0.5.1     
WebOb                     1.8.7     
Werkzeug                  2.2.2     
zipp                      3.11.0    
zope.deprecation          4.4.0     
zope.interface            5.5.2     
zope.sqlalchemy           1.6

Wim-De-Clercq commented 1 year ago

The pip and setuptools versions are too old (those are from 2020), and not good enough to deal with clashing dependencies.

The (reduced) dependency tree you end up with, with the old versions is this:

atramhasis==1.3.0
  - openapi-spec-validator [required: 0.4.0, installed: 0.4.0]
  - pyramid-openapi3 [required: Any, installed: 0.15.0]
    - openapi-core [required: >=0.16.1, installed: 0.16.5]
      - openapi-spec-validator [required: >=0.5.0,<0.6.0, installed: 0.4.0]

Which is clearly wrong because the openapi-spec-validator requirement from openapi-core does not match with what was installed.

With a modern pip and setuptools the dependency resolver is smart enough to realise this and use older versions which do match. If you install the dependencies after you do a pip install --upgrade pip setuptools you should end up with

atramhasis==1.3.0
  - openapi-spec-validator [required: 0.4.0, installed: 0.4.0]
  - pyramid-openapi3 [required: Any, installed: 0.14.3]
    - openapi-core [required: >=0.13.4,<0.14, installed: 0.13.8]
      - openapi-spec-validator [required: Any, installed: 0.4.0]

which is valid.

koenedaele commented 1 year ago

Documentation was updated. References to virtualenv have been removed. Instructions now include upgrading setuptools and pip.

OnroerendErfgoed / atramhasis