search

tony-framework / TonY

TonY is a framework to natively run deep learning frameworks on Apache Hadoop.
https://tony-project.ai
Other
708 stars 164 forks source link

Can the training task run with docker container, and the hadoop cluster run without docker container? #601

Closed jxfruit closed 2 years ago

jxfruit commented 3 years ago

May I have a question about docker mode? Can the training task run with docker container, and the hadoop cluster run without docker container?

zuston commented 3 years ago

@jxfruit Of course. If want to training on docker, the higher hadoop version should be needed, like hadoop3.x.

Can you describe more details?

jxfruit commented 3 years ago

OK，training codes and tony client will be packaged running on docker, which cannot depend on the docker containerized hadoop cluster, right? Can u provide some docs? And are there some examples about my issue? tks a lot

zuston commented 3 years ago

Doc is here. Any question can be issued here.

jxfruit commented 3 years ago

@zuston the docs is not very clear. And yarn should run on docker ?

zuston commented 3 years ago

@jxfruit TonY now support two options to run tensorflow/pytorch jobs.

  1. python venv
  2. using the docker which is depended on Hadoop3.x supporting docker runtime

Could you describe more details on your cluster information like Hadoop version? If you want to know more about the TonY examples or best practices, please send to my email: junfan.zhang@outlook.com. Maybe we could chat offline

jxfruit commented 3 years ago

@zuston Thanks a lot. And I have sent an email.

oliverhu commented 3 years ago

@zuston you can also discuss in the TonY slack channel so the knowledge can be shared..