Fates

[annefou]

• need to add an example
• be able to set namelist paramters in "user_nl_clm". The paremeters that can be interesting for users are: use_fates (if false, the model will be CLM. The rest of the parameters can only be used when "use_fates" is .true.)
• fates_spitfire_mode
• use_fates_planthydro
• use_fates_cohort_age_tracking
• use_fates_ed_st3
• use_fates_ed_prescribed_phys
• use_fates_logging
• use_fates_inventory_init (not working without a inventory file, so not needed for the moment. )
• use_fates_fixed_biogeog
• fates_parteh_mode

doc https://github.com/NordicESMhub/ctsm/blob/fates_emerald_api/bld/namelist_files/namelist_definition_ctsm.xml

• add an image with site location?

[annefou]

Should .travis.yml be updated? I thought automated testing was turned on for this repository.

[bgruening]

Travis changes some settings I think. You need to look at the travis page and maybe migrate something. I don't think you need to update the file - not for this but we need to adopt it later for python3.

I also think that a .shed.yml file is missing, correct?

[annefou]

Travis changes some settings I think. You need to look at the travis page and maybe migrate something. I don't think you need to update the file - not for this but we need to adopt it later for python3.

We get an error with travis:

Collecting pycodestyle<2.7.0,>=2.6.0a1
  Downloading pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB)
Requirement already satisfied: typing; python_version < "3.5" in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from flake8) (3.6.6)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from flake8) (0.18)
Requirement already satisfied: setuptools in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from flake8-import-order) (44.1.1)
Collecting BeautifulSoup4
  Downloading beautifulsoup4-4.9.3-py2-none-any.whl (115 kB)
ERROR: Could not find a version that satisfies the requirement bioblend>=0.14.0 (from planemo) (from versions: 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.6.0, 0.6.1, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0)
ERROR: No matching distribution found for bioblend>=0.14.0 (from planemo)
The command "pip install flake8 flake8-import-order planemo" failed and exited with 1 during .
Your build has been stopped.

Do we need to change from python 2 to python 3?

[annefou]

travis is now failing because its takes more than 10mn. The suggestion at https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received is to add some more time: install: travis_wait 30 mvn install but honestly I am not sure where to add it (it is failing with planemo test:

HOME=`pwd` && mkdir -p .cime && cp '/home/travis/.planemo/planemo_tmp_nlm0a2km/config' .cime/config && cp '/home/travis/.planemo/planemo_tmp_nlm0a2km/config_compilers.xml' '.cime/config_compilers.xml' && cp '/home/travis/.planemo/planemo_tmp_nlm0a2km/config_machines.xml' '.cime/config_machines.xml' && mkdir inputdata && tar -xf '/tmp/tmp5_z1rgav/files/e/1/c/dataset_e1cf33a8-dfc2-46f3-8a56-e8cd0d1a3bf6.dat' -C 'inputdata/' && mkdir output_dir && mkdir usermods_dirs && create_newcase --case 'test' --compset 2000_DATM%1PTGSWP3_CLM50%FATES_SICE_SOCN_MOSART_SGLC_SWAV --machine espresso --run-unsupported --res '1x1_ALP1'  && cd 'test' && ./case.setup  > ../output_dir/case.txt  2>&1   && ./xmlchange STOP_N=5 && ./xmlchange STOP_OPTION=ndays && ./xmlchange CCSM_CO2_PPMV=367.0 && ./xmlchange DOUT_S=FALSE && cp '/tmp/tmp5_z1rgav/job_working_directory/000/2/configs/tmpc_d1lvkv' user_nl_clm && ./case.build >> ../output_dir/case.txt  2>&1 && ./case.submit >> ../output_dir/case.txt  2>&1 && mkdir -p restart                              && (cat '../work/test/run/rpointer'.* > /tmp/tmp5_z1rgav/files/6/7/e/dataset_67e682ce-9ad7-4417-bd1f-10ef136c638e.dat || printf "No restarts\n") && (cp '../work/test/run/test'.*.r*.*.nc restart/ || printf "No netCDF restarts\n") && (cp '../work/test/run/test'.*.r*.*.bin restart/ || printf "No bin restarts\n") && (cp '../work/test/run/rpointer'.* restart/ || printf "no rpointer\n") && cd restart && tar cvf ../../output_dir/restart.tar . && cd .. && (cp '../work/test/run/test'.*.h*.*.nc ../output_dir/ || printf "no netCDF outputs\n") && cp '../work/test/run/'*.log.* ../output_dir/ || (printf "Case failed" && tar cf ../output_dir/work.tar ../work)]
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

[bgruening]

@annefou if you can provide me temporary admin access, I will try to set something different up.

[annefou]

@annefou if you can provide me temporary admin access, I will try to set something different up.

Admin access to the github repo?

[bgruening]

Yes.

[annefou]

Yes.

you should be now (I am not sure you get a notification for this kind of changes). A big thank you for your help!

[bgruening]

Hi Anne.

This is now using GitHub actions. The new stuff that we are using in IUC and elsewhere. It makes certain things more robust, but also more strict. But it can run for 4-5h.

You need to set the TS_API_KEY under your github settings for this repo, to enable to push to the TSs.

One other advantage is that you can now download the planemo summary from github. For example see this one https://github.com/galaxyproject/tools-iuc/actions/runs/310913713 ... if you click on "All tool test results" you can download an HTML report.

I hope you like it.

[annefou]

I hope you like it.

Yes I love it!!! just need to understand where to define these variables and to what... Thank you very much!!!

[bgruening]

Here: https://github.com/NordicESMhub/galaxy-tools/settings/secrets and to your ToolShed API keys.

[bgruening]

@annefou one thing (that makes it more complicated) is that this is now testing your tool in containers by default. Your tools are special :) as they ship a compiler and so on. If this makes too much headache we can disable the container testing and use conda as before.

[annefou]

@annefou one thing (that makes it more complicated) is that this is now testing your tool in containers by default. Your tools are special :) as they ship a compiler and so on. If this makes too much headache we can disable the container testing and use conda as before.

If it works then it is great!

[annefou]

Here: https://github.com/NordicESMhub/galaxy-tools/settings/secrets and to your ToolShed API keys.

I feel very stupid but I can't remember where to get these infos (about toolshed climate...). I thought I could use what we had already defined for the repo. Is there a way to find out what I used...

[annefou]

Here: https://github.com/NordicESMhub/galaxy-tools/settings/secrets and to your ToolShed API keys.

I feel very stupid but I can't remember where to get these infos (about toolshed climate...). I thought I could use what we had already defined for the repo. Is there a way to find out what I used..

I think I found the password...

[bgruening]

Here are the first test-results https://github.com/NordicESMhub/galaxy-tools/suites/1361167159/artifacts/22198814 (not sure why it is testing more than fates, I guess because of my crazy merges.)

The API key you can find here https://toolshed.g2.bx.psu.edu/ --> User --> API key and in the testtoolshed

[annefou]

Here are the first test-results https://github.com/NordicESMhub/galaxy-tools/suites/1361167159/artifacts/22198814 (not sure why it is testing more than fates, I guess because of my crazy merges.)

The API key you can find here https://toolshed.g2.bx.psu.edu/ --> User --> API key and in the testtoolshed

I added a key for Copernicus so hopefully it will fix the essential climate variable tool.

[annefou]

Looking at the error for FATES, it looks like the tool needs to know the value of the environment variable USER so I added a key. Should we rerun it?

[bgruening]

Looking at the error for FATES, it looks like the tool needs to know the value of the environment variable USER so I added a key. Should we rerun it?

Can this be a real problem? In a conainer the user might not exist. Maybe we can set a fake user as part of the cheetah tool-xml?

[annefou]

Can this be a real problem? In a conainer the user might not exist. Maybe we can set a fake user as part of the cheetah tool-xml?

At least this is one of them... What you are saying is that it is probably not sufficient to define USER as a github key so we should define it in the xml file. Is it correct?

[bgruening]

Yes. Not sure why this USER is needed in the first place.

[annefou]

Here are the first test-results https://github.com/NordicESMhub/galaxy-tools/suites/1361167159/artifacts/22198814

How do you get this link? Is it always the same link?

[bgruening]

You see it above under "actions": https://github.com/NordicESMhub/galaxy-tools/actions/runs/315226636

[annefou]

@bgruening I am a bit stuck here; I don't fully understand. From the errors I get, it looks like environment variables (such as GALAXY_COPERNICUS_CDSAPIRC_KEY) are not defined while I added it in https://github.com/NordicESMhub/galaxy-tools/settings/secrets Should I do anything else?

[bgruening]

@annefou I can take care of this tool. Please feel free to concentrate on Fates ... I suspect that those envs are not passed into the container.

[annefou]

@annefou I can take care of this tool. Please feel free to concentrate on Fates ... I suspect that those envs are not passed into the container.

Thanks! I think for fates, we have the same problem with USER not being passed too; which is stranged because I also defined it in the xml too.

[bgruening]

@annefou fates is now passing the tests ....

[annefou]

@annefou fates is now passing the tests ....

Yes!!!!!!!!!! Thanks!!! I still need to work on the outputs as you mentioned earlier that one collection with mixed content/formats is not the way to go. And I am not very familiar with collections so I don't even know how to use them in the test section!

[bgruening]

What I usually do is searching in IUC :) (this time on github with collection and test)

This is a nice example maybe? https://github.com/galaxyproject/tools-iuc/blob/7f7605c2c8d8e92f3369dfdc290e5d8d06fa409a/tools/mothur/classify.otu.xml#L123

Otherwise planemo has some nice collection documentation and examples as well on readthedocs.

[annefou]

Is there a way to get the outputs generated? Now it does not do anything and I can't find out why. If I could get the outputs I would get some hints

[annefou]

@bgruening I removed the ecv tool so everything is now green!