Issues building the pipeline

[jds485]

Documenting issues for PR #101: I'm using conda 4.12.0

1. Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.

   • I think you can solve this by adding pip to the dependencies list

2. Solving environment: failed
   ResolvePackageNotFound:

   • snakemake-minimal
   • torchvision
   • torchaudio
   • plotly=5.5.0

• This is probably due to different default channels that you all have. Here are mine: channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch

[galengorski]

Thanks for trying this @jds485 , comments below and more to come

I'm using conda 4.12.0

I am using conda 4.12.0 as well

I think you can solve this by adding pip to the dependencies list

I removed the dependency on the netcdf module, I think that wasn't being used anymore. This was the only pip-installed module, so I also removed pip. I've updated the environment.yaml file within #101

Solving environment: failed ResolvePackageNotFound: snakemake-minimal torchvision torchaudio plotly=5.5.0 This is probably due to different default channels that you all have. Here are mine: channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch

Mine are: https://repo.anaconda.com/pkgs/main https://repo.anaconda.com/pkgs/r https://repo.anaconda.com/pkgs/msys2

I need to look deeper into how to resolve this, will it build if you comment out those packages? Looking at my conda list those packages have different channels: snakemake-minimal : bioconda torchaudio: pytorch torchvisual: pytorch plotly=5.5.0: plotly

Do those channels need to be installed first...?

[jds485]

I can add those channels to the list in the environment.yaml file and see if it builds correctly. I think specifying the exact channels to use will help with reproducibility and shareability

[jds485]

It gets past these package not found errors when I add these channels: channels:

• defaults
• bioconda
• pytorch
• plotly

There are package conflicts that it's trying to resolve now. I'll let you know if it doesn't succeed

[jds485]

Pushed updated environment file in #101

[jds485]

Running during the conference talks. I haven't looked into this, but would appreciate suggestions: Snakefile_fetch_munge: Missing input files for rule all_fetch_munge: affected files: 03a_it_analysis/out/srcs_proc_lagged 03a_it_analysis/out/snks_proc

[galengorski]

Hmm, I think there might be a couple of holdover files from before we split the snakemake files. Try this: comment out the first two files in rule all_fetch_munge so it should look like this:

[galengorski]

I'm running this in parallel to you in a newly created environment, and it's working. I'll try to see if I can reproduce any errors you might get

[galengorski]

Part of the issues that we were running into were caused by the fact that the newest version of the snakemake file was not in #101 and so @jds485 didn't have the latest version, my fault. I have merged the latest version into the main branch. I then transferred the fetch_munge snakefile to #101 along with a few changes to the environment.yaml file. Here are steps that I think should get this running:

1. pull down the latest changes from #101 git fetch origin pull/101/head:03b_modeling_steps_cleaner
2. update the environment conda env update --file environment.yaml
3. check to make sure sciencebasepy is loaded into the environment using conda list. (I could not get sciencebasepy to load via the yaml file, I have some ideas but I think this is a separate discussion)
4. If sciencebasepy is not listed in conda list then pip install sciencebasepy
5. Now you should be ready to run snakemake. snakemake -s Snakefile_fetch_munge --cores all