Command pip install -e . failing

mielvds commented 1 month ago

Hello! i'm unable to build the latest atramhasis from source. I'm using a docker build located here: https://github.com/viaacode/atramhasis-docker.

It's hard to tell what's going on. It's somehow stuck on the Grunt install process, which recently became part of the pip install -e . process? (I used to install the JS only after the python install; it's still in the dockerfile). Any ideas?

 => ERROR [ 9/17] RUN pip install -e .                                                                                                                                                                                                   12.1s 
------                                                                                                                                                                                                                                         
 > [ 9/17] RUN pip install -e .:                                                                                                                                                                                                               
0.382 Obtaining file:///app/atramhasis                                                                                                                                                                                                         
0.384   Installing build dependencies: started                                                                                                                                                                                                 
1.197   Installing build dependencies: finished with status 'done'                                                                                                                                                                             
1.197   Checking if build backend supports build_editable: started                                                                                                                                                                             
1.221   Checking if build backend supports build_editable: finished with status 'done'
1.221   Getting requirements to build editable: started
1.265   Getting requirements to build editable: finished with status 'done'
1.267   Installing backend dependencies: started
1.731   Installing backend dependencies: finished with status 'done'
1.732   Preparing editable metadata (pyproject.toml): started
11.68   Preparing editable metadata (pyproject.toml): finished with status 'error'
11.69   error: subprocess-exited-with-error
11.69   
11.69   × Preparing editable metadata (pyproject.toml) did not run successfully.
11.69   │ exit code: -15
11.69   ╰─> [100 lines of output]
11.69       
11.69       added 3 packages, and audited 6 packages in 560ms
11.69       
11.69       found 0 vulnerabilities
11.69       npm warn skipping integrity check for git dependency ssh://git@github.com/stratease/dGrowl.git
11.69       
11.69       added 121 packages, and audited 122 packages in 9s
11.69       
11.69       7 packages are looking for funding
11.69         run `npm fund` for details
11.69       
11.69       found 0 vulnerabilities
11.69       Initializing
11.69       Command-line options: --verbose, --gruntfile=/app/atramhasis/atramhasis/static/admin/Gruntfile.js
11.69       
11.69       Reading "Gruntfile.js" Gruntfile...OK
11.69       
11.69       Registering Gruntfile tasks.
11.69       Initializing config...OK
11.69       
11.69       Registering "grunt-contrib-copy" local Npm module tasks.
11.69       Reading /app/atramhasis/atramhasis/static/admin/node_modules/grunt-contrib-copy/package.json...OK
11.69       Parsing /app/atramhasis/atramhasis/static/admin/node_modules/grunt-contrib-copy/package.json...OK
11.69       Loading "copy.js" tasks...OK
11.69       + copy
11.69       
11.69       Registering "grunt-contrib-clean" local Npm module tasks.
11.69       Reading /app/atramhasis/atramhasis/static/admin/node_modules/grunt-contrib-clean/package.json...OK
11.69       Parsing /app/atramhasis/atramhasis/static/admin/node_modules/grunt-contrib-clean/package.json...OK
11.69       Loading "clean.js" tasks...OK
11.69       + clean
11.69       
11.69       Registering "grunt-dojo" local Npm module tasks.
11.69       Reading /app/atramhasis/atramhasis/static/admin/node_modules/grunt-dojo/package.json...OK
11.69       Parsing /app/atramhasis/atramhasis/static/admin/node_modules/grunt-dojo/package.json...OK
11.69       Loading "dojo.js" tasks...OK
11.69       + dojo
11.69       Loading "Gruntfile.js" tasks...OK
11.69       + build
11.69       
11.69       Running tasks: build
11.69       
11.69       Running "build" task
11.69       
11.69       Running "clean" task
11.69       
11.69       Running "clean:dist" (clean) task
11.69       Verifying property clean.dist exists in config...OK
11.69       Files: [no src]
11.69       Options: force=false, no-write=false
11.69       >> 0 paths cleaned.
11.69       
11.69       Running "dojo:dist" (dojo) task
11.69       Verifying property dojo.dist exists in config...OK
11.69       File: [no files]
11.69       Options: dojo="node_modules/dojo/dojo.js", load="build", profile="profiles/atramhasis.profile.js", profiles=null, appConfigFile=null, package=null, packages=null, require=null, requires=null, releaseDir="../dist", action=null, cwd=null, dojoConfig="src/dojoConfig.js", basePath="/app/atramhasis/atramhasis/static/admin/node_modules"
11.69       
11.69       Building Dojo...
11.69       Dojo build parameters:
11.69           node_modules/dojo/dojo.js
11.69           load=build
11.69           --basePath /app/atramhasis/atramhasis/static/admin/node_modules
11.69           --profile profiles/atramhasis.profile.js
11.69           --profile profiles/atramhasis.profile.js
11.69           --dojoConfig src/dojoConfig.js
11.69           --releaseDir ../dist
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/profiles/atramhasis.profile.js
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/profiles/atramhasis.profile.js
11.69       processing dojoConfig resource /app/atramhasis/atramhasis/static/admin/src/dojoConfig.js
11.69       info(107) Package Version: package: dijit; version: 1.17.3
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dijit/dijit.profile.js
11.69       info(107) Package Version: package: dojo; version: 1.17.3
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dojo/dojo.profile.js
11.69       info(107) Package Version: package: dgrid; version: 1.2.1
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dgrid/package.js
11.69       info(107) Package Version: package: dstore; version: 1.2.1
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/profiles/atramhasis.profile.js
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/profiles/atramhasis.profile.js
11.69       processing dojoConfig resource /app/atramhasis/atramhasis/static/admin/src/dojoConfig.js
11.69       info(107) Package Version: package: dijit; version: 1.17.3
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dijit/dijit.profile.js
11.69       info(107) Package Version: package: dojo; version: 1.17.3
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dojo/dojo.profile.js
11.69       info(107) Package Version: package: dgrid; version: 1.2.1
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dgrid/package.js
11.69       info(107) Package Version: package: dstore; version: 1.2.1
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dstore/package.js
11.69       info(107) Package Version: package: dGrowl; version: 0.1.2
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dGrowl/package.js
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/src/app/package.js
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dstore/package.js
11.69       info(107) Package Version: package: dGrowl; version: 0.1.2
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/node_modules/dGrowl/package.js
11.69       processing profile resource /app/atramhasis/atramhasis/static/admin/src/app/package.js
11.69       discovering resources...
11.69       starting reading resources...
11.69       starting processing raw resource content...
11.69       starting tokenizing resource...
11.69       starting processing resource tokens...
11.69       starting parsing resource...
11.69       [end of output]
11.69   
11.69   note: This error originates from a subprocess, and is likely not a problem with pip.
11.70 error: metadata-generation-failed
11.70 
11.70 × Encountered error while generating package metadata.
11.70 ╰─> See above for output.
11.70 
11.70 note: This is an issue with the package mentioned above, not pip.
11.70 hint: See above for details.

goessebr commented 1 month ago

Hi @mielvds,

I'm assuming that you are using version 2.1.0a2. Note that this is still an alpha release, but let's try to find out what is going wrong here.

It's somehow stuck on the Grunt install process, which recently became part of the pip install -e . process?

Correct. The frontend installs are included via a build hook: build_hook.py. The hook is executed when building a wheel or sdist. When using the cookiebuilders to build a new Atramhasis scaffold application, we can use the built frontend from the wheel or sdist, but when using Atramhasis itself, an (automatic) frontend build is still needed.

Furthermore, you can no longer use python setup.py compile_catalog. Instead, you can use:

pybabel compile --directory 'atramhasis/locale' --domain atramhasis --statistics true

I will make the documentation clearer about this before I make the final release. The general installation should be pretty accurate, but the admin development section needs a huge update.

The Dockerfile no longer needs:

Lines 25-29

# install js dependencies for public site using npm
RUN cd atramhasis/static && npm install
# install js dependencies for admin using npm
RUN cd atramhasis/static/admin && npm install
#&& grunt -v build

Lines 40
```
#&& grunt -v build
```

Lines 23-24

# compile the Message Catalog Files
RUN python setup.py compile_catalog

I don't think the error will be fixed just by updating your Dockerfile with the changes above. I suppose the error has something to do with the version of the installers. These are my versions. If you are using a lower version, you might want to upgrade. If you are using a higher version, we probably need to adjust the Atramhasis code.

npm --version
10.2.4
grunt --version
grunt-cli v1.4.3
node --version
v21.6.2

To pinpoint the problem, instead of running pip install -e ., you can check out the 2.1.0a2 release (or the develop branch) and build the frontend code with these steps:

cd <atramhasis_dir>/atramhasis/static
npm install
cd <atramhasis_dir>/atramhasis/static/admin
npm install
cd <atramhasis_dir>/atramhasis/static/admin
grunt -v build

Does this provide clearer feedback on the error?

koenedaele commented 1 month ago

Is there actually a reason to clone the Github repo for your Docker file? If you're trying to create your own instance, it would make more sense to use the "create your own instance" cookiecutter (https://atramhasis.readthedocs.io/en/latest/customisation.html#creating-your-own-project). This creates a custom python package you can use to hold your own custom templates and css and when a new Atramhasis is released you generally only need to update the requirements file and possibly update your own overrides. This way the Atramhasis admin environment is built for you already so you don't need to build it yourself (and don't need the build tools). At least that's the way we envisioned it being used when we built the software. If it doesn't work in practice, we'd love to get some feedback on it and see if we can improve it or if we're better of just scrapping this whole way of working.

goessebr commented 4 weeks ago

I would like to add that if you are using the cookiecutters as documented in https://atramhasis.readthedocs.io/en/latest/customisation.html#creating-your-own-project, you will need to specifically install the latest atramhasis alpha version.

pip install -e . will install all requirements, including atramhasis. But instead of the alpha version the latest stable version (2.0.0) will be installed. Please make sure that before executing pip install -e . you install the atramhasis alpha version via pip install atramhasis==2.1.0a2 as long as a stable 2.1.0 is not released. This way atramhasis is already present in your virtualenv and the installer will not download the stable version.

mielvds commented 3 weeks ago

Hi @koenedaele @goessebr good point! When I created this docker repo, I wasn't sure whether the cookiecutter was up to speed with atramhasis 2.x. But indeed, this does work! Check out https://github.com/viaacode/atramhasis-docker/blob/main/meemoo_atramhasis/Dockerfile.

Here are some more insights by running the individual commands with the original Dockerfile, where grunt -v build seems to cause the issue:

0.373 >> dojo build error: Error
0.375 Warning: Task "dojo:dist" failed. Use --force to continue.
0.375 
0.375 Aborted due to warnings.
------
Dockerfile:21
--------------------
  19 |     RUN cd atramhasis/static && npm install
  20 |     RUN cd atramhasis/static/admin && npm install 
  21 | >>> RUN cd atramhasis/static/admin && grunt -v build
  22 |     # create or update database
  23 |     RUN alembic upgrade head
--------------------
ERROR: failed to solve: process "/bin/sh -c cd atramhasis/static/admin && grunt -v build" did not complete successfully: exit code: 3

goessebr commented 3 weeks ago

Hi @mielvds,

What is your output for grunt --version?

mielvds commented 3 weeks ago

grunt-cli v1.4.3 grunt v1.6.1

Wim-De-Clercq commented 2 weeks ago

It's of course not easy to tell what's wrong without seeing the actual process output. But you do get that error in the Task "dojo:dist" phase when you don't have java available, which grunt uses internally.

mielvds commented 2 weeks ago

It was indeed the missing Java JDK! The demo site now builds.

It's still a mystery to me why the demosite (Dockerfile) needs Java and the cookiecutter output scaffold (Dockerfile) does not. The latter is either not building the admin site or it's no longer using grunt...

Wim-De-Clercq commented 2 weeks ago

Your demosite docker clones the project from github, and has to then build everything, including frontend. Which is the grunt and java build.

The scaffold downloads atramhasis from pypi. Which will give you a prebuilt wheel file which contains the frontend already built, so no grunt or java runs.

OnroerendErfgoed / atramhasis

Command pip install -e . failing #881