Updating GLM pipeline with new lakes

DOI-USGS / lake-temperature-process-models-old

Pipeline #2

Other

0 stars 6 forks source link

Updating GLM pipeline with new lakes #8

Closed jzwart closed 4 years ago

jzwart commented 5 years ago

Resurrecting this pipeline for our lake temperature sprint.

Updated lake list and shapefiles from pipeline #1
Set up makefile for generating nml files for running GLM. I don't think it makes sense to create an ind file for these so am just creating these in a tmp dir, which will get rsync-ed to Yeti.
Copied over some Yeti functions that Alison created. These should help with automatically syncing the pipeline to Yeti. Updated README with some instructions on how to set up ssh with Yeti.

jzwart commented 5 years ago

Based on building the ~100 nml files, it should take about 95 mins to generate all ~19,000 nml files on one computer locally

limnoliver commented 5 years ago

@jzwart in the past we've ignored task remake files. It looks like Alison also does this in pipeline three using a similar combiner model. I think you'll want to add 1_nml_tasks_makefile.yml to .gitignore?

jordansread commented 5 years ago

Agreed w/ Sam on the task makefiles, especially if they are this big. (I think it is fine to include small ones if it makes sense to).

jzwart commented 5 years ago

Looks good to me, but wondering which scipiper version are you using? Is it the one w/ the task_combiners? or previous version?

I'm a little unclear if I'm using the combiner version of scipiper. I updated this morning but with the master branch. Looks like there is task_combiner branch of scipiper, which I am not using

jordansread commented 5 years ago

@limnoliver what are you using?

I am using the combiner branch for my work on the temperature prep pipeline...I think we should all use the same version, but I understand that the jump to the combiners version is not trivial (hence it is still hanging out on a branch).

If you are willing to make the jump, it will help us simplify some of the solutions and also help us from colliding w/ different versions on different parts of the pipeline.

limnoliver commented 5 years ago

I was using the latest version of the master branch. But yes, I think we should switch to the combiners branch since @jzwart is using that workflow as well.

jzwart commented 5 years ago

Sounds good, I'll switch to the combiners branch

jordansread commented 4 years ago

Sorry, I didn't see that this was still open until now. I am working on a similar pattern, but now that we have nhd HR, some of the functions won't work (e.g., populate_base_nml())