search

lucasmsp / docker-atlas

Cluster in docker with Apache Atlas and a minimal Hadoop ecosystem to perform some basic experiments.
Apache License 2.0
25 stars 26 forks source link

Can't run spark #2

Closed ZSLajevardi closed 1 year ago

ZSLajevardi commented 2 years ago

Hi I've been trying to install apache atlas on ubuntu 20.04 (local) using this docker image, but it has been stuck on step 12 while running spark.

Screenshot (82)

Any ideas what went wrong here?

lucasmsp commented 2 years ago

Hi, Could you provide more information? Did you change any variable in Dockerfile like SPARK_VERSION or HADOOP_VERSION ? Probably this error is due to tar was not able to extract the spark tgz file, probably by a problem in curl like an unavailable link or any other network problem. If you did not change Spark version, the Dockerfile will try to download spark at https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz. You are able to download it manually?

ZSLajevardi commented 2 years ago

Hey there, Thanks for your responding. I did not change anything and I just simply run 'docker compose up' after cloning the repository and after pulling wurstmeister/kafka , wbaa/rokku-dev-apache-atlas, bde2020/hive-metastore-postgresql, wurstmeister/zookeeper, maven, bde2020/hadoop-datanode , bde2020/hadoop-namenode and bde2020/hive; I have the following results:

Screenshot (86)

Screenshot (87)

And in the end, yes I can download spark manually from https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz but it took about 15-20 minutes to download.

lucasmsp commented 2 years ago

Hi, I guess it is a bug in your network or your docker version. If you want to debug further, try to download spark using this link in a created ubuntu container. If is bug in docker, trying to compile the following cd spark; docker build --network=host -t lucasmsp/spark:2.4 . could help (by using the host network, Docker's DNS problems are avoided).

Anyway, I just update this repository to use a more recent Ubuntu image. In addition, I update docker-compose file to use an already compiled image (docker pull lucasmsp/spark:2.4). I hope this helps you.

ZSLajevardi commented 2 years ago

Hi @lucasmsp I used the updated repository and the spark error is resolved. but after I received the 'Apache Atlas server Started' massage, I've got 503 error as following:

ocker-atlas-atlas-server-1 | ............................................................................... docker-atlas-atlas-server-1 | Apache Atlas Server started!!! docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | Creating ingestion-source type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed 23 1529 100 354 0 0 1076 0HTTP/1.1 503 Service Unavailable0 docker-atlas-atlas-server-1 | Date: Sat, 16 Apr 2022 04:23:04 GMT docker-atlas-atlas-server-1 | Cache-Control: must-revalidate,no-cache,no-store docker-atlas-atlas-server-1 | Content-Type: text/html;charset=iso-8859-1 docker-atlas-atlas-server-1 | Content-Length: 354 docker-atlas-atlas-server-1 | Connection: close docker-atlas-atlas-server-1 | Server: Jetty(9.3.14.v20161028) docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | Error 503 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 |

HTTP ERROR: 503

docker-atlas-atlas-server-1 |

Problem accessing /api/atlas/v2/types/typedefs. Reason: docker-atlas-atlas-server-1 | 

    Service Unavailable

docker-atlas-atlas-server-1 |
Powered by Jetty:// 9.3.14.v20161028
docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | --:--:-- --:--:-- --:--:-- 1079 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | ingestion-source created docker-atlas-atlas-server-1 | Creating file type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed 100 1012 100 354 100 658 12936 24045 --:--:-- --:--:-- --:--:-- 24370 docker-atlas-atlas-server-1 | HTTP/1.1 503 Service Unavailable docker-atlas-atlas-server-1 | Date: Sat, 16 Apr 2022 04:23:04 GMT docker-atlas-atlas-server-1 | Cache-Control: must-revalidate,no-cache,no-store docker-atlas-atlas-server-1 | Content-Type: text/html;charset=iso-8859-1 docker-atlas-atlas-server-1 | Content-Length: 354 docker-atlas-atlas-server-1 | Server: Jetty(9.3.14.v20161028) docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | Error 503 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 |

HTTP ERROR: 503

docker-atlas-atlas-server-1 |

Problem accessing /api/atlas/v2/types/typedefs. Reason: docker-atlas-atlas-server-1 | 

    Service Unavailable

docker-atlas-atlas-server-1 |
Powered by Jetty:// 9.3.14.v20161028
docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | classifications type created docker-atlas-atlas-server-1 | Creating rokku_client type... docker-atlas-atlas-server-1 | % Total % Received % Xferd Average Speed Time Time Time Current docker-atlas-atlas-server-1 | Dload Upload Total Spent Left Speed 20 HTTP/1.1 503 Service Unavailable0 0 --:--:-- --:--:-- --:--:-- 0 docker-atlas-atlas-server-1 | Date: Sat, 16 Apr 2022 04:23:04 GMT docker-atlas-atlas-server-1 | Cache-Control: must-revalidate,no-cache,no-store docker-atlas-atlas-server-1 | Content-Type: text/html;charset=iso-8859-1 docker-atlas-atlas-server-1 | Content-Length: 354 docker-atlas-atlas-server-1 | Connection: close docker-atlas-atlas-server-1 | Server: Jetty(9.3.14.v20161028) docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | Error 503 docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 |

HTTP ERROR: 503

docker-atlas-atlas-server-1 |

Problem accessing /api/atlas/v2/types/typedefs. Reason: docker-atlas-atlas-server-1 | 

    Service Unavailable

docker-atlas-atlas-server-1 |
Powered by Jetty:// 9.3.14.v20161028
docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | docker-atlas-atlas-server-1 | rokku_client created docker-atlas-atlas-server-1 | 1687 100 354 0 0 16833 0 --:--:-- --:--:-- --:--:-- 17700 docker-atlas-atlas-server-1 | Done setting up Atlas types

I can manually access 'http://eclipse.org/jetty' but It got stuck here and the process won't end unless I gracefully cancel it.

lucasmsp commented 2 years ago

Hi, starting Atlas and others containers (hive, spark, etc) at the same time could take some time depending on your machine. Although the errors, I can see through this log that atlas is progressing. Try to start only atlas and its dependencies first (docker-compose up atlas-server zookeeper kafka). The first start will take a longer time because it needs to download and prepare the environment.