search

bio-guoda / guoda-services

Services provided by GUODA, currently a container for tickets and wikis.
MIT License
2 stars 0 forks source link

hdfs instable #57

Closed jhpoelen closed 5 years ago

jhpoelen commented 5 years ago

Various re-occurring issues (like https://github.com/bio-guoda/guoda-services/issues/54 https://github.com/gimmefreshdata/freshdata/issues/85#issuecomment-433969920) indicate that HDFS (our distributed file system) is unable to deal with the load. Anecdotal evidence suggests that FreshData's updateAll job is creating a long lasting heavy load causing hdfs to crash.

@mjcollin @jhammock @diatomsRcool fyi

jhammock commented 5 years ago

I see action was taken on #59 , but this ticket is still open. Should I try UpdateMonitors again or no?

jhpoelen commented 5 years ago

Looks like effechecka.org and freshdata are behaving nowadays. @jhammock please close issue if you agree.

jhammock commented 5 years ago

I'm mostly in the dark here, but I have been waiting all day for a makeparquet task- a symptom which was, i think, my first clue to this problem last time...

jhpoelen commented 5 years ago

Please open a separate issue on that and include logging. Please note that only a single job is currently processed at the time and Anne's checklist scripts is sending jobs automatically.

You can test hdfs access by adding / removing a file in /from hdfs on the command line.

jhammock commented 5 years ago

adding files works fine. I moved the iNat resource into a temp directory with no trouble. Please define "include logging".

jhpoelen commented 5 years ago

about logging - the thing you see when you submit the make parquet script / job