Build datahub-gms failed

[ddsr-ops]

Firstly, I succeed in starting project using datahub docker quickstart --build-locally , output as follows:

datahub-gms is up-to-date
Starting mysql-setup         ... 
schema-registry is up-to-date
Starting elasticsearch-setup ... done
Starting mysql-setup         ... done
Starting kafka-setup         ... done
........
✔ DataHub is now running
Ingest some demo data using `datahub docker ingest-sample-data`,
or head to http://localhost:9002 (username: datahub, password: datahub) to play around with the frontend.
Need support? Get in touch on Slack: https://slack.datahubproject.io/
(venv) [root@centos7 smoke-test]# docker ps -a
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS                     PORTS                                              NAMES
9c634a54559d        linkedin/datahub-kafka-setup:head           "/bin/sh -c ./kafk..."   4 minutes ago       Exited (0) 2 minutes ago                                                      kafka-setup
179088f1d2f9        confluentinc/cp-schema-registry:5.4.0       "/etc/confluent/do..."   4 minutes ago       Up 4 minutes               0.0.0.0:8081->8081/tcp                             schema-registry
3d0f2f6421be        confluentinc/cp-kafka:5.4.0                 "/etc/confluent/do..."   4 minutes ago       Up 4 minutes               0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   broker
1a7e972ca910        linkedin/datahub-elasticsearch-setup:head   "/bin/sh -c 'if [ ..."   4 minutes ago       Exited (0) 2 minutes ago                                                      elasticsearch-setup
a3c3b5a5592d        linkedin/datahub-frontend-react:head        "datahub-frontend/..."   4 minutes ago       Up 4 minutes (healthy)     0.0.0.0:9002->9002/tcp                             datahub-frontend-react
29b00faf02df        acryldata/acryl-datahub-actions:head        "/bin/sh -c 'docke..."   4 minutes ago       Up 4 minutes                                                                  datahub_datahub-actions_1
add51808d6fb        linkedin/datahub-gms:head                   "/bin/sh -c /datah..."   4 minutes ago       Up 4 minutes (healthy)     0.0.0.0:8080->8080/tcp                             datahub-gms
4012a8b94e06        acryldata/datahub-mysql-setup:head          "/bin/sh -c 'docke..."   4 minutes ago       Exited (0) 2 minutes ago                                                      mysql-setup
ec8a8105c78e        confluentinc/cp-zookeeper:5.4.0             "/etc/confluent/do..."   4 minutes ago       Up 4 minutes               2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp         zookeeper
1b24cbc307c3        elasticsearch:7.9.3                         "/tini -- /usr/loc..."   4 minutes ago       Up 4 minutes (healthy)     0.0.0.0:9200->9200/tcp, 9300/tcp                   elasticsearch
60d76ab5ad50        mysql:5.7                                   "docker-entrypoint..."   4 minutes ago       Up 3 minutes               0.0.0.0:3306->3306/tcp, 33060/tcp                  mysql

However, I failed to start gms container by using (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate datahub-gms), output as follows:

[root@centos7 datahub]# docker -v
Docker version 1.13.1, build 7d71120/1.13.1
[root@centos7 datahub]# docker-compose -v
docker-compose version 1.29.2, build 5becea4c
[root@centos7 datahub]# (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate datahub-gms)
Building datahub-gms
unknown flag: --iidfile
See 'docker build --help'.
ERROR: Service 'datahub-gms' failed to build : Build failed

Thanks for your help.

[RyanHolstien]

Hi! Have you built the code locally prior to trying to bring up the server? Please see: https://datahubproject.io/docs/docker/development/ for the typical steps on how to start up locally.

[ddsr-ops]

@RyanHolstien Yes, I have built the project according to https://datahubproject.io/docs/developers, but some tests failed i ignored them.

315 tests completed, 6 failed, 250 skipped

> Task :docs-website:yarnBuild
✔ Client: Compiled successfully in 22.27s

> Task :datahub-web-react:yarnLint
Done in 200.39s.

> Task :docs-website:yarnBuild
✔ Server: Compiled successfully in 32.83s
Success! Generated static files in "build".

Use `npm run serve` command to test your build locally.

Done in 141.09s.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':metadata-io:test'.
> There were failing tests. See the report at: file:///root/datahub/metadata-io/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 5m 1s

Bring up the docker according to https://datahubproject.io/docs/developers.

[root@centos7 datahub]# ls
build.gradle                 datahub.iml         docker             gradlew              metadata-dao-impl            metadata-jobs              metadata-testing  repositories.gradle
CODEOWNERS                   datahub.ipr         docs               gradlew.bat          metadata-events              metadata-models            metadata-utils    settings.gradle
contrib                      datahub.iws         docs-website       ingestion-scheduler  metadata-ingestion           metadata-models-custom     node_modules      smoke-test
datahub-frontend             datahub-kubernetes  entity-registry    LICENSE              metadata-ingestion-examples  metadata-models-validator  NOTICE            temp_time_series
datahub-gms-graphql-service  datahub-upgrade     gradle             li-utils             metadata-integration         metadata-perf              perf-test         test-models
datahub-graphql-core         datahub-web-react   gradle.properties  lombok.config        metadata-io                  metadata-service           README.md         yarn.lock
[root@centos7 datahub]# (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate datahub-gms)
Creating network "datahub_network" with the default driver
Creating volume "datahub_esdata" with default driver
Creating volume "datahub_zkdata" with default driver
Creating volume "datahub_mysqldata" with default driver
Building datahub-gms
unknown flag: --iidfile
See 'docker build --help'.
ERROR: Service 'datahub-gms' failed to build : Build failed

Thank you for your reply.

[ddsr-ops]

Maybe docker engine version is incompatible with docker-compose version?

[RyanHolstien]

Maybe docker engine version is incompatible with docker-compose version?

Possibly, looks like someone else ran into this in version 1.29 here: https://stackoverflow.com/questions/67010408/unable-to-start-docker-container-from-docker-compose-unknown-flag-iidfile so you could try downgrading and see if that changes things.

but that is the version I have locally as well (1.29.2) and I have no issues running the command.

Yes, I have built the project according to https://datahubproject.io/docs/developers, but some tests failed i ignored them.

Do you mean you re-ran the build after this with the tests marked as @ Ignore in the code and the build ran to completion afterwards? If the output files from the build are not present then the development mode start up will not work properly.

[ddsr-ops]

@RyanHolstien I downgrade the docker-compose, now the version is 1.26.2. I restart the project , output as follows:

[root@centos7 datahub]# (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate datahub-gms)
WARNING: Native build is an experimental feature and could change at any time
Building datahub-gms
unknown flag: --iidfile
See 'docker build --help'.
Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 1078, in up
  File "compose/cli/main.py", line 1074, in up
  File "compose/project.py", line 548, in up
  File "compose/service.py", line 367, in ensure_image_exists
  File "compose/service.py", line 1110, in build
  File "compose/progress_stream.py", line 25, in stream_output
  File "compose/utils.py", line 61, in split_buffer
  File "compose/utils.py", line 37, in stream_as_text
  File "compose/service.py", line 1819, in build
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpb9lzn2iv'

[RyanHolstien]

Have you followed the docker instructions for installing on CentOS here?: https://docs.docker.com/engine/install/centos/ Or how has docker been installed on the machine?

[ddsr-ops]

Have you followed the docker instructions for installing on CentOS here?: https://docs.docker.com/engine/install/centos/ Or how has docker been installed on the machine?

I will try to reinstall docker following https://docs.docker.com/engine/install/centos/. Thank you .

[ddsr-ops]

I referenced https://github.com/datahub-project/datahub/issues/2020 to cover it. Docker version is 1.13.1, and docker-compose version is 1.27.4.