search

carpentries-lab / python-aos-lesson

Python for Atmosphere and Ocean Scientists
https://carpentries-lab.github.io/python-aos-lesson/
Other
87 stars 48 forks source link

Add content on data management #4

Closed DamienIrving closed 6 years ago

DamienIrving commented 6 years ago

The content on data reference syntax from my old capstone lesson might be useful here?

DamienIrving commented 6 years ago

What do I do if cdo and nco aren't available via git bash after installing them with conda?

annefou commented 6 years ago

nco is available in conda and if you use anaconda navigator, you can use open a Terminal (with all the necessary paths according you installed it with anaconda).

cdo is not available on windows (only very old version). What you can do with cdo can be done with other tools available on Windows. The performance is just not very good... The alternative we have investigated was to use a docker container. For learning tools it is very handy. Docker is easy to install and works well except on old systems, especially Windows 7. Using docker on Windows 10 is smooth and easy.

The main problem we face here is to learn to use tools installed on our laptop while it is not really the path we recommend for processing large amount of data. I have been thinking working on a lesson using google cloud or Amazon cloud.

DamienIrving commented 6 years ago

Thanks, @annefou. I wasn't aware of the issue with cdo on windows. I notice that conda-forge doesn't have a cdo package for windows at all, not even for the very old version: https://anaconda.org/search?q=cdo

I agree regarding the laptop issue - we spend all this time trying to figure out work-arounds for people on windows laptops, but the reality is that very few (if any) people in the atmosphere and ocean sciences actually do their real work on a windows machine. (As evidenced by the fact that cdo - one of the most popular tools in the field - doesn't even work on windows).

I wonder if the best solution is to have participants work in the environment that they actually do their work in? For most people, I'd imagine that would mean bringing along their laptop and ssh-ing into a remote (linux) machine at their home institution. This means we can be 100% sure that when they leave the workshop, they can apply what they've learned in their own work.

The great thing about conda is that you don't need admin access to install it - it's just a directory that you put somewhere on your system. If we have everyone install miniconda instead of the full anaconda installation (and use the command line for package management instead of the navigator), then we can be sure that the lessons will work for everyone (i.e. then we don't have to worry about X11 forwarding to get the navigator to work via ssh).