search

lucasmsp / docker-atlas

Cluster in docker with Apache Atlas and a minimal Hadoop ecosystem to perform some basic experiments.
Apache License 2.0
25 stars 26 forks source link

Apache Atlas Distribution 3.0.0-SNAPSHOT failure #5

Closed davidpr closed 1 year ago

davidpr commented 1 year ago

Hi, when incrementing the resources in Docker desktop to 12GB RAM and 4GB Swap the process of building goes farer but it get an error in the last service to be built.

0 745.6 [INFO] Apache Atlas classification updater ................ SUCCESS [ 0.508 s]

0 745.6 [INFO] Apache Atlas Impala Hook API ....................... SUCCESS [ 0.083 s]

0 745.6 [INFO] Apache Atlas Impala Bridge Shim .................... SUCCESS [ 0.098 s]

0 745.6 [INFO] Apache Atlas Impala Bridge ......................... SUCCESS [ 2.773 s]

0 745.6 [INFO] Apache Atlas Distribution 3.0.0-SNAPSHOT ........... FAILURE [05:23 min]

0 745.6 [INFO] ------------------------------------------------------------------------

0 745.6 [INFO] BUILD FAILURE

0 745.6 [INFO] ------------------------------------------------------------------------

0 745.6 [INFO] Total time: 12:09 min

0 745.6 [INFO] Finished at: 2022-12-07T09:11:58Z

0 745.6 [INFO] ------------------------------------------------------------------------

0 745.6 [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-test) on project atlas-distro: Execution python-test of goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec failed: Plugin org.codehaus.mojo:exec-maven-plugin:1.2.1 or one of its dependencies could not be resolved: Failed to collect dependencies at org.codehaus.mojo:exec-maven-plugin:jar:1.2.1 -> org.apache.maven:maven-toolchain:jar:1.0 -> org.apache.maven:maven-core:jar:2.0.6 -> org.apache.maven:maven-plugin-api:jar:2.0: Failed to read artifact descriptor for org.apache.maven:maven-plugin-api:jar:2.0: Could not transfer artifact org.apache.maven:maven-plugin-api:pom:2.0 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]

0 745.6 [ERROR]

0 745.6 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

0 745.6 [ERROR] Re-run Maven using the -X switch to enable full debug logging.

0 745.6 [ERROR]

0 745.6 [ERROR] For more information about the errors and possible solutions, please read the following articles:

0 745.6 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

0 745.6 [ERROR]

0 745.6 [ERROR] After correcting the problems, you can resume the build with the command

0 745.6 [ERROR] mvn -rf :atlas-distro

failed to solve: process "/bin/sh -c git clone http://github.com/apache/atlas.git \t&& cd atlas \t&& git reset --hard $ATLAS_REV \t&& mvn clean -DskipTests package -Pdist,embedded-hbase-solr \t&& mv distro/target/apache-atlas-*-bin.tar.gz /apache-atlas.tar.gz" did not complete successfully: exit code: 1

lucasmsp commented 1 year ago

Hi @davidpr, the error message is related to Maven, a tool used by Atlas (and many other big projects) to download its dependencies and compile them as a single project. The error "Connection reset" means that Maven was unable to download some jars while it was trying to build Atlas. The Maven repository was probably slow or your network was having some problems at the moment. Try it again. Moreover, if you already have a valid and compiled docker image, you dont need to rebuild when resources are being incremented.

davidpr commented 1 year ago

Hi @lucasmsp , thanks for you answer. I've tried several times but I get the same error. I'll try with cable connection just in case. By the way, if after getting this error I just run the docker-compose up directly I get the appliance running. Although I don't know if this can be done or the installation will be missing something even though it's running.

I have these messages after having the Apache Atlas server running:

kafka | [2022-12-07 09:46:46,816] INFO [GroupCoordinator 1001]: Assignment received from leader for group atlas for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) docker-atlas-atlas-server-1 | Creating ingestion-source type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed 100 3080 0 1905 100 1175 88 54 0:00:21 0:00:21 --:--:-- 159 docker-atlas-atlas-server-1 | HTTP/1.1 100 Continue docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | HTTP/1.1 200 OK docker-atlas-atlas-server-1 | Date: Wed, 07 Dec 2022 09:46:55 GMT docker-atlas-atlas-server-1 | Set-Cookie: ATLASSESSIONID=1au3q82emxbuxw0v149erjrrl;Path=/;HttpOnly docker-atlas-atlas-server-1 | Expires: Thu, 01 Jan 1970 00:00:00 GMT docker-atlas-atlas-server-1 | X-Frame-Options: DENY docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | X-XSS-Protection: 1; mode=block docker-atlas-atlas-server-1 | Strict-Transport-Security: max-age=31536000; includeSubDomains docker-atlas-atlas-server-1 | Content-Type: application/json; charset=UTF-8 docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data: docker-atlas-atlas-server-1 | Server: Apache Atlas docker-atlas-atlas-server-1 | Transfer-Encoding: chunked docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | {"enumDefs":[],"structDefs":[],"classificationDefs":[{"category":"CLASSIFICATION","guid":"2fca3ac7-f806-453d-a1fe-d580af512110","createdBy":"admin","updatedBy":"admin","createTime":1670406415296,"updateTime":1670406415296,"version":1,"name":"staging_node","description":"Used for classifying a data scientist WS","typeVersion":"1.0","attributeDefs":[],"superTypes":[],"entityTypes":[],"subTypes":[]},{"category":"CLASSIFICATION","guid":"7c287761-69a5-45ac-9ae6-75b48eda4692","createdBy":"admin","updatedBy":"admin","createTime":1670406415307,"updateTime":1670406415307,"version":1,"name":"storage_node","description":"Used for classifying a data backend","typeVersion":"1.0","attributeDefs":[],"superTypes":[],"entityTypes":[],"subTypes":[]}],"entityDefs":[{"category":"ENTITY","guid":"7d3f93ad-bab3-4e9e-b5f2-b0f8b84946ab","createdBy":"admin","updatedBy":"admin","createTime":1670406415315,"updateTime":1670406415315,"version":1,"name":"server","description":"a type definition for server machine","typeVersion":"1.0","attributeDefs":[{"name":"server_name","typeName":"string","isOptional":false,"cardinality":"SINGLE","valuesMinCount":1,"valuesMaxCount":1,"isUnique":true,"isIndexable":true,"includeInNotification":false,"searchWeight":-1},{"name":"ip_address","typeName":"string","isOptional":false,"cardinality":"SINGLE","valuesMinCount":1,"valuesMaxCount":1,"isUnique":true,"isIndexable":true,"includeInNotification":false,"searchWeight":-1}],"superTypes":["Infrastructure"],"subTypes":[],"relationshipAttributeDefs":[{"name":"meanings","typeName":"array","isOptional":true,"cardinality":"SET","valuesMinCount":-1,"valuesMaxCount":-1,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1,"relationshipTypeName":"AtlasGlossarySemanticAssignment","isLegacyAttribute":false}],"businessAttributeDefs":{}}],"relationshipDefs":[],"businessMetadataDefs":[]} docker-atlas-atlas-server-1 | ingestion-source created docker-atlas-atlas-server-1 | Creating file type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed HTTP/1.1 200 OK 0 100 658 0 65 0:00:10 0:00:10 --:--:-- 0 docker-atlas-atlas-server-1 | Date: Wed, 07 Dec 2022 09:47:16 GMT docker-atlas-atlas-server-1 | Set-Cookie: ATLASSESSIONID=v4q83yigpk8z154nj9nivd046;Path=/;HttpOnly docker-atlas-atlas-server-1 | Expires: Thu, 01 Jan 1970 00:00:00 GMT docker-atlas-atlas-server-1 | X-Frame-Options: DENY docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | X-XSS-Protection: 1; mode=block docker-atlas-atlas-server-1 | Strict-Transport-Security: max-age=31536000; includeSubDomains docker-atlas-atlas-server-1 | Content-Type: application/json; charset=UTF-8 docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data: docker-atlas-atlas-server-1 | Server: Apache Atlas docker-atlas-atlas-server-1 | Transfer-Encoding: chunked docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | {"enumDefs":[],"structDefs":[],"classificationDefs":[{"category":"CLASSIFICATION","guid":"28f82290-e71a-4511-98b7-ba0d3295111e","createdBy":"admin","updatedBy":"admin","createTime":1670406436713,"updateTime":1670406436713,"version":1,"name":"customer_PII","description":"Used for classifying a data which contains customer personal information, hence indicating confidential private","typeVersion":"1.0","attributeDefs":[],"superTypes":[],"entityTypes":[],"subTypes":[]},{"category":"CLASSIFICATION","guid":"c1a94b53-f7b5-4096-b715-15ee5464445a","createdBy":"admin","updatedBy":"admin","createTime":1670406436724,"updateTime":1670406436724,"version":1,"name":"customer_NON_PII","description":"Used for classifying a data which contains customer non personal information, hence indicating confidential lesser than private","typeVersion":"1.0","attributeDefs":[],"superTypes":[],"entityTypes":[],"subTypes":[]}],"entityDefs":[],"relationshipDefs":[],"businessMetadataDefs":[]}100 1632 0 974 100 658 95 64 0:00:10 0:00:10 --:--:-- 76 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | classifications type created docker-atlas-atlas-server-1 | Creating rokku_client type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed 100 4009 0 2676 100 1333 127 63 0:00:21 0:00:21 --:--:-- 332HTTP/1.1 100 Continue docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | HTTP/1.1 200 OK docker-atlas-atlas-server-1 | Date: Wed, 07 Dec 2022 09:47:26 GMT docker-atlas-atlas-server-1 | Set-Cookie: ATLASSESSIONID=kflltb2h68wk19vi2sy9zrrd1;Path=/;HttpOnly docker-atlas-atlas-server-1 | Expires: Thu, 01 Jan 1970 00:00:00 GMT docker-atlas-atlas-server-1 | X-Frame-Options: DENY docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | X-XSS-Protection: 1; mode=block docker-atlas-atlas-server-1 | Strict-Transport-Security: max-age=31536000; includeSubDomains docker-atlas-atlas-server-1 | Content-Type: application/json; charset=UTF-8 docker-atlas-atlas-server-1 | X-Content-Type-Options: nosniff docker-atlas-atlas-server-1 | Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data: docker-atlas-atlas-server-1 | Server: Apache Atlas docker-atlas-atlas-server-1 | Transfer-Encoding: chunked docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | {"enumDefs":[],"structDefs":[],"classificationDefs":[],"entityDefs":[{"category":"ENTITY","guid":"60928b33-5e20-4329-bda9-c3bf76b960ed","createdBy":"admin","updatedBy":"admin","createTime":1670406446869,"updateTime":1670406446869,"version":1,"name":"rokku_client","description":"a type definition for AWS tools","typeVersion":"1.1","attributeDefs":[{"name":"operation","typeName":"string","isOptional":true,"cardinality":"SINGLE","valuesMinCount":0,"valuesMaxCount":1,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1},{"name":"run_as","typeName":"string","isOptional":false,"cardinality":"SINGLE","valuesMinCount":1,"valuesMaxCount":1,"isUnique":false,"isIndexable":true,"includeInNotification":false,"searchWeight":-1}],"superTypes":["Process"],"subTypes":[],"relationshipAttributeDefs":[{"name":"outputs","typeName":"array","isOptional":true,"cardinality":"SET","valuesMinCount":0,"valuesMaxCount":2147483647,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1,"relationshipTypeName":"process_dataset_outputs","isLegacyAttribute":true},{"name":"server","typeName":"server","isOptional":true,"cardinality":"SINGLE","valuesMinCount":-1,"valuesMaxCount":-1,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1,"relationshipTypeName":"rokku_client_server","isLegacyAttribute":true},{"name":"inputs","typeName":"array","isOptional":true,"cardinality":"SET","valuesMinCount":0,"valuesMaxCount":2147483647,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1,"relationshipTypeName":"dataset_process_inputs","isLegacyAttribute":true},{"name":"meanings","typeName":"array","isOptional":true,"cardinality":"SET","valuesMinCount":-1,"valuesMaxCount":-1,"isUnique":false,"isIndexable":false,"includeInNotification":false,"searchWeight":-1,"relationshipTypeName":"AtlasGlossarySemanticAssignment","isLegacyAttribute":false}],"businessAttributeDefs":{}}],"relationshipDefs":[{"category":"RELATIONSHIP","guid":"ea434df8-7a04-40c1-8b19-f17cd7edc81f","createdBy":"admin","updatedBy":"admin","createTime":1670406446882,"updateTime":1670406446882,"version":1,"name":"rokku_client_server","description":"rokku_client_server","typeVersion":"1.0","attributeDefs":[],"relationshipCategory":"ASSOCIATION","relationshipLabel":"__rokku_client.server","propagateTags":"NONE","endDef1":{"type":"server","name":"clients","isContainer":false,"cardinality":"SET","isLegacyAttribute":true},"endDef2":{"type":"rokku_client","name":"server","isContainer":false,"cardinality":"SINGLE","isLegacyAttr100 4009 0 2676 100 1333 127 63 0:00:21 0:00:21 --:--:-- 442 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | rokku_client created docker-atlas-atlas-server-1 | Done setting up Atlas types docker-atlas-namenode-1 | 22/12/07 10:18:23 INFO BlockStateChange: BLOCK* processReport 0x374ba966d5c: from storage DS-938abe26-58e6-46e6-a8a3-f4341898fd55 node DatanodeRegistration(172.19.0.4:50010, datanodeUuid=da385388-7d8b-4f44-accd-e5d1ad8f919b, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-76b745d7-b701-44bc-8883-8699c4bbb1b0;nsid=952075926;c=0), blocks: 0, hasStaleStorage: false, processing time: 98 msecs docker-atlas-datanode-1 | 22/12/07 10:18:23 INFO datanode.DataNode: Successfully sent block report 0x374ba966d5c, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 6 msec to generate and 146 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5. docker-atlas-datanode-1 | 22/12/07 10:18:23 INFO datanode.DataNode: Got finalize command for block pool BP-1660066545-172.19.0.6-1670355708823