Cannot build project with docker

[Mnikley]

Building the docker-compose.yml from both, the master as well as the simple-docker branch, fails.

Background information

• OS: Windows 11 Version 23H2 (OS Build 22631.3296)
• Docker version 24.0.6, build ed223bc
• Docker Compose version v2.23.0-desktop.1

Steps to reproduce

1. Clone repository

   git clone https://github.com/biocypher/meta-graph.git

2. Switch to simple-docker branch

   git checkout simple-docker
   Switched to a new branch 'simple-docker'
   branch 'simple-docker' set up to track 'origin/simple-docker'.

3. build stack

   docker-compose up -d
   [+] Building 96.6s (13/16)                                                                                                                                           docker:default
   => [neo4j internal] load .dockerignore                                                                                                                                        0.0s
   => => transferring context: 88B                                                                                                                                               0.0s
   => [neo4j internal] load build definition from Dockerfile                                                                                                                     0.0s
   => => transferring dockerfile: 802B                                                                                                                                           0.0s
   => [neo4j internal] load metadata for docker.io/library/neo4j:4.4-enterprise                                                                                                  0.0s
   => [neo4j internal] load metadata for docker.io/andimajore/biocyper_base:python3.10                                                                                           1.9s
   => [neo4j setup-stage 1/7] FROM docker.io/andimajore/biocyper_base:python3.10@sha256:fe3c33ff90142a10dc8f03b711e5ed9551b989dff2b23b31e3abeb24afa9dc65                        82.5s
   => => resolve docker.io/andimajore/biocyper_base:python3.10@sha256:fe3c33ff90142a10dc8f03b711e5ed9551b989dff2b23b31e3abeb24afa9dc65                                           0.0s
   => => sha256:713e36be6fa3e32dd6f30c398d8a0f2dd5424740d09c6c1d8233f85a314e73aa 7.28kB / 7.28kB                                                                                 0.0s
   => => sha256:a64cfb0db31fea25d4887162ae68fabd569a4ed82352dafcf808d6b0d037e46e 26.68MB / 26.68MB                                                                               1.5s
   => => sha256:ebe13c243416e6967f1eb718abf176f5a0b162736a292ed23ce9dff176a43e77 193B / 193B                                                                                     0.5s
   => => sha256:6f9aad1a896c2653b3c48017290ae66901bbc66a0f282f2c50ef30f6f7f1c023 190.67MB / 190.67MB                                                                            14.5s
   => => sha256:fe3c33ff90142a10dc8f03b711e5ed9551b989dff2b23b31e3abeb24afa9dc65 4.33kB / 4.33kB                                                                                 0.0s
   => => sha256:9cbacfe1d0bf437b19b75bc398ebd614f09a627b52cbbeea4772f98db2ff3fce 411B / 411B                                                                                     1.0s
   => => sha256:6440b0129bb7be8cd445e952c3889277903d896f33d1aad4ee4b09d5dae1f1d0 232.85MB / 232.85MB                                                                            12.1s
   => => sha256:8db8f24076f98acfdfeb150aa9f77ae7bbb0b8b893595a6ece4ea3030880b2fb 1.74kB / 1.74kB                                                                                 2.1s
   => => extracting sha256:a64cfb0db31fea25d4887162ae68fabd569a4ed82352dafcf808d6b0d037e46e                                                                                      2.0s
   => => sha256:55cb12e8634cb3d1036219b766d4a8ba6bb6adffeff4618bdb7af333c01fb772 148.12MB / 148.12MB                                                                            30.7s
   => => extracting sha256:ebe13c243416e6967f1eb718abf176f5a0b162736a292ed23ce9dff176a43e77                                                                                      0.0s
   => => sha256:7e9a8ecce3d78a0be0b11c3ab100de54c488ba92b0aecdfb183f29dea81a7c28 85.16MB / 85.16MB                                                                              15.9s
   => => sha256:749a0cda1d90127d567d537186c88abb0a8b219394781acbf2ad0b3b6849b2e3 4.18MB / 4.18MB                                                                                15.6s
   => => extracting sha256:6f9aad1a896c2653b3c48017290ae66901bbc66a0f282f2c50ef30f6f7f1c023                                                                                     16.6s
   => => sha256:ff4baaee41e16d407ddfb6d6cf05086eb72b0c175d6614fcdfa43300be1bf5dc 8.74MB / 8.74MB                                                                                18.7s
   => => sha256:4997f937190e146fc6613c452cb573f8d4daf29abf4304289f4c63a57883acc0 409.58MB / 409.58MB                                                                            42.5s
   => => sha256:0bd53e37461213b6ada6dac5c9bcba5bc5fc7f8638a57a73b3e5446e14715f41 26.94MB / 26.94MB                                                                              23.6s
   => => sha256:af611935963ec7b2cc1e613bd1ebf47c690a09a5218fb44e78d7f9c0d6172bce 121.29MB / 121.29MB                                                                            30.2s
   => => sha256:47c1b5aa04b022346419527937713c9f136c0cc015d7249335042c48d480dd26 102.22MB / 102.22MB                                                                            39.3s
   => => sha256:e83d958854976bfa49f63c51352d7996a059fa45f2c36c5e1dd23ca89eb04bc7 3.08MB / 3.08MB                                                                                32.0s
   => => extracting sha256:9cbacfe1d0bf437b19b75bc398ebd614f09a627b52cbbeea4772f98db2ff3fce                                                                                      0.0s
   => => extracting sha256:6440b0129bb7be8cd445e952c3889277903d896f33d1aad4ee4b09d5dae1f1d0                                                                                     12.4s
   => => sha256:beeb5d0c6fe61396270905f5578a43b593672f59009e52370c80d93d7d55179c 194.43MB / 194.43MB                                                                            41.8s
   => => sha256:381aefa9f778c952abe09aaaa81546cbce8a0246a410c7a346d66fa9ac50294e 195.86MB / 195.86MB                                                                            48.4s
   => => sha256:bc4d5258b1c9300fb1787147998e40f7099f2365d6baf2b2e93a71443fc77a8d 155B / 155B                                                                                    42.5s
   => => sha256:6aa38a1d25081d6d67477f924a6ccb6a4816439bd3043fa2f9a1b6144a6521eb 31.95MB / 31.95MB                                                                              45.2s
   => => extracting sha256:8db8f24076f98acfdfeb150aa9f77ae7bbb0b8b893595a6ece4ea3030880b2fb                                                                                      0.0s
   => => extracting sha256:55cb12e8634cb3d1036219b766d4a8ba6bb6adffeff4618bdb7af333c01fb772                                                                                      6.2s
   => => extracting sha256:7e9a8ecce3d78a0be0b11c3ab100de54c488ba92b0aecdfb183f29dea81a7c28                                                                                      2.7s
   => => extracting sha256:749a0cda1d90127d567d537186c88abb0a8b219394781acbf2ad0b3b6849b2e3                                                                                      0.3s
   => => extracting sha256:ff4baaee41e16d407ddfb6d6cf05086eb72b0c175d6614fcdfa43300be1bf5dc                                                                                      0.2s
   => => extracting sha256:4997f937190e146fc6613c452cb573f8d4daf29abf4304289f4c63a57883acc0                                                                                     13.9s
   => => extracting sha256:0bd53e37461213b6ada6dac5c9bcba5bc5fc7f8638a57a73b3e5446e14715f41                                                                                      0.4s
   => => extracting sha256:af611935963ec7b2cc1e613bd1ebf47c690a09a5218fb44e78d7f9c0d6172bce                                                                                      4.0s
   => => extracting sha256:47c1b5aa04b022346419527937713c9f136c0cc015d7249335042c48d480dd26                                                                                      3.6s
   => => extracting sha256:e83d958854976bfa49f63c51352d7996a059fa45f2c36c5e1dd23ca89eb04bc7                                                                                      0.2s
   => => extracting sha256:beeb5d0c6fe61396270905f5578a43b593672f59009e52370c80d93d7d55179c                                                                                      1.4s
   => => extracting sha256:381aefa9f778c952abe09aaaa81546cbce8a0246a410c7a346d66fa9ac50294e                                                                                      2.5s
   => => extracting sha256:bc4d5258b1c9300fb1787147998e40f7099f2365d6baf2b2e93a71443fc77a8d                                                                                      0.0s
   => => extracting sha256:6aa38a1d25081d6d67477f924a6ccb6a4816439bd3043fa2f9a1b6144a6521eb                                                                                      1.2s
   => [neo4j deploy-stage 1/4] FROM docker.io/library/neo4j:4.4-enterprise                                                                                                       0.1s
   => [neo4j internal] load build context                                                                                                                                        0.1s
   => => transferring context: 59.33kB                                                                                                                                           0.0s
   => [neo4j setup-stage 2/7] WORKDIR /usr/app/                                                                                                                                  1.1s
   => [neo4j setup-stage 3/7] COPY pyproject.toml poetry.lock ./                                                                                                                 0.0s
   => [neo4j setup-stage 4/7] RUN poetry config virtualenvs.create false && poetry install                                                                                       9.2s
   => [neo4j setup-stage 5/7] COPY . ./                                                                                                                                          0.0s
   => [neo4j setup-stage 6/7] RUN cp config/biocypher_docker_config.yaml config/biocypher_config.yaml                                                                            0.4s
   => ERROR [neo4j setup-stage 7/7] RUN python3 create_knowledge_graph.py                                                                                                        1.4s
   ------
   > [neo4j setup-stage 7/7] RUN python3 create_knowledge_graph.py:
   0.505 INFO -- This is BioCypher v0.5.9.
   0.505 INFO -- Logging into `biocypher-log/biocypher-20240408-070734.log`.
   1.315 Traceback (most recent call last):
   1.315   File "/usr/app/create_knowledge_graph.py", line 35, in <module>
   1.315     adapter = BioCypherMetaAdapter(
   1.315   File "/usr/app/meta_graph/adapters/adapter.py", line 63, in __init__
   1.315     self._download_data()
   1.315   File "/usr/app/meta_graph/adapters/adapter.py", line 99, in _download_data
   1.315     id_ = self._get_project_id(url, headers)
   1.315   File "/usr/app/meta_graph/adapters/adapter.py", line 129, in _get_project_id
   1.315     return data.get("organization").get("projectV2").get("id")
   1.315 AttributeError: 'NoneType' object has no attribute 'get'
   ------
   failed to solve: process "/bin/sh -c python3 create_knowledge_graph.py" did not complete successfully: exit code: 1

[slobentanzer]

Hi @Mnikley, thanks for the report, and apologies for the lack of transparency. The simple-docker requires a valid API key in the environment and in general is an older experimental branch. The main branch is the one that should work. I added some info in the README of the simple-docker branch to clarify this.

As for the main branch, for me this works fine with

git clone https://github.com/biocypher/meta-graph.git
cd meta-graph
docker compose up -d

Can you make sure to remove all remaining Docker components from previous runs (the volumes can be a problem if they are not removed properly), using docker compose down --volumes?

[Mnikley]

Hey @slobentanzer,

thank you for your reply. Using the main branch helped me to progress, thank you. Eventually I could start the stack after fixing the line endings of the bash scripts scripts/build.sh and scripts/import.sh, which were causing an issue due to cloning the repository on Windows with autocrlf enabled (git config --get core.autocrlf -> true):

1. Cleared old stack, checked out the main branch, tried to start the stack:

   docker-compose down
   git checkout main
   docker-compose up -d

   time="2024-04-16T09:16:24+02:00" level=warning msg="The \"BIOCYPHER_GITHUB_PROJECT_TOKEN\" variable is not set. Defaulting to a blank string."
   [+] Building 0.0s (0/0)                                                                                                                                              docker:default
   [+] Running 5/5
   ✔ Network meta-graph_default                     Created                                                                                                                      0.0s
   ✔ Volume "meta-graph_biocypher-meta-graph_data"  Created                                                                                                                      0.0s
   ✘ Container build                                service "build" didn't complete successfully: exit 1                                                                         0.0s
   ✔ Container import                               Created                                                                                                                      0.0s
   ✔ Container deploy                               Created                                                                                                                      0.0s
   service "build" didn't complete successfully: exit 1

   docker-compose logs

   time="2024-04-16T09:16:29+02:00" level=warning msg="The \"BIOCYPHER_GITHUB_PROJECT_TOKEN\" variable is not set. Defaulting to a blank string."
   build  | /src/scripts/build.sh: line 2: cd: $'/usr/app/\r': No such file or directory
   build  | cp: target '.'$'\r' is not a directory
   build  | cp: cannot stat 'config/biocypher_docker_config.yaml': No such file or directory
   build  |
   " does not exist.and "install
   build  |
   build  | Did you mean this?
   build  |     install
   build  | python3: can't open file '/app/create_knowledge_graph.py\r': [Errno 2] No such file or directory
   build  | chmod: cannot access 'biocypher-log': No such file or directory

2. Switched to WSL and fixed line endings of bash scripts with dos2unix

   dos2unix scripts/build.sh
   dos2unix scripts/import.sh

3. Cleared & restarted stack, and it worked

Also I was wondering if you deliberately use your image v1.0.0 in the docker-compose file, as I saw that you already released versions 1.1.0 and 1.1.1 ?

Thanks again, Matthias

[slobentanzer]

Hi Matthias, thanks for the explanation, I unfortunately do not work on Windows, and these path things are somewhat of a mystery to me. :)

Do you think this should be explained somewhere, or should we even take some measures to prevent this from happening for other Windows users? @nilskre FYI