yarn-site.xml and yarn-env.sh missing from etc/hadoop configuration directory

[he-sk]

The build instructions suggest to replace etc/hadoop with sample_conf and adapt it to the local machine. However, to figure out what has changed, it is useful to run a diff of sample_conf against a pristine configuration directory. However, when replacing hadoop-yarn-project, the files yarn-site.xml and yarn-env.sh are missing from etc/hadoop.

[giagulei]

Hi Viktor,

I am not sure I understood the last part of your question. When exactly are these files missing?

G.

Στις Παρ, 21 Ιουν 2019 στις 7:41 μ.μ., ο/η Viktor Rosenfeld < notifications@github.com> έγραψε:

The build instructions suggest to replace etc/hadoop with sample_conf and adapt it to the local machine. However, to figure out what has changed, it is useful to run a diff of sample_conf against a pristine configuration directory. However, when replacing hadoop-yarn-project, the files yarn-site.xml and yarn-env.sh are missing from etc/hadoop.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/E2Data/E2DataYARN/issues/1?email_source=notifications&email_token=AAS3JWS67NMTY4WKWLMOK5TP3UAFNA5CNFSM4H2THAI2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G3ADI2A, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS3JWTNX3PDQCKBFFCK7J3P3UAFNANCNFSM4H2THAIQ .

[he-sk]

Hi @giagulei,

I replaced hadoop-yarn-project with E2DataYARN and build from sources. However, I did not replace etc/hadoop with sample_conf. Instead, I'm trying to reproduce the necessary changes to figure out why the NodeManager is not coming up. But when I unpack the hadoop distribution after build, these files are missing from etc/hadoop.

[he-sk]

I noticed this because I get the following error:

viktor@gpu-4:~/hadoop/e2data-changes/hadoop-3.1.2$ sbin/start-dfs.sh
ERROR: Invalid HADOOP_YARN_HOME

But adding the files does not fix it.

[giagulei]

Ok, I 've never noticed such a thing. I will look into it and get back to you probably on Monday.

Thanks, Giagkos

Στις Παρ, 21 Ιουν 2019 στις 7:47 μ.μ., ο/η Viktor Rosenfeld < notifications@github.com> έγραψε:

Hi @giagulei https://github.com/giagulei,

I replaced hadoop-yarn-project with E2DataYARN and build from sources. However, I did not replace etc/hadoop with sample_conf. Instead, I'm trying to reproduce the necessary changes to figure out why the DataNode is not coming up. But when I unpack the hadoop distribution after build, these files are missing from etc/hadoop.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/E2Data/E2DataYARN/issues/1?email_source=notifications&email_token=AAS3JWSV3EISIIY6FTLKFDDP3UA3TA5CNFSM4H2THAI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYI7NAY#issuecomment-504493699, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS3JWR7BJI7IV5YIDBYVCLP3UA3TANCNFSM4H2THAIQ .

[giagulei]

Viktor can you also send me the configuration you are using for starting YARN?

Στις Παρ, 21 Ιουν 2019 στις 7:51 μ.μ., ο/η Mytilinis Giagos < gmytil@cslab.ece.ntua.gr> έγραψε:

Ok, I 've never noticed such a thing. I will look into it and get back to you probably on Monday.

Thanks, Giagkos

Στις Παρ, 21 Ιουν 2019 στις 7:47 μ.μ., ο/η Viktor Rosenfeld < notifications@github.com> έγραψε:

Hi @giagulei https://github.com/giagulei,

I replaced hadoop-yarn-project with E2DataYARN and build from sources. However, I did not replace etc/hadoop with sample_conf. Instead, I'm trying to reproduce the necessary changes to figure out why the DataNode is not coming up. But when I unpack the hadoop distribution after build, these files are missing from etc/hadoop.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/E2Data/E2DataYARN/issues/1?email_source=notifications&email_token=AAS3JWSV3EISIIY6FTLKFDDP3UA3TA5CNFSM4H2THAI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYI7NAY#issuecomment-504493699, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS3JWR7BJI7IV5YIDBYVCLP3UA3TANCNFSM4H2THAIQ .

[he-sk]

Okay, I just noticed that I'm missing anything related to YARN. For example, sbin/start-yarn.sh.

However, I also noticed that the E2DataYARN is based on Hadoop 3.1.1 and I tried to install it into 3.1.2. I'll try to fix this.

[he-sk]

The root cause was that I tried to compile this repo (which is based on Hadoop 3.1.1) in Hadoop 3.1.2.