'Module Not Found' Error running fastStructure

[quinn-ca]

I went down a long rabbithole trying to get FastStructure to work, which ultimately led me to trying to use structure_threader. I'm working in a conda virtual environment (remote linux server) with python3 installed.

I installed the faststructure dependencies and added gsl to my path, per fastStructure and then installed structure_threader using pip.

However, when I run my code, I get two errors: 'ModuleNotFoundError: No module named 'plotter' and 'ModuleNotFoundError: No module named 'colorlover''.

I think these modules are specific to structure_threader and so should have installed with it? My code (a slurm job) is below and I sincerely appreciate your help!

#!/bin/bash
#
#SBATCH --job-name=structurethreader_K1to5_autosomal_genstr_rmRelated.sh
#SBATCH --output=structurethreader_K1to5_autosomal_genstr_rmRelated.txt
#SBATCH --error=structurethreader_K1to5_autosomal_genstr_rmRelated.error
#SBATCH --cpus-per-task=25
#SBATCH --time=100:00:00
#SBATCH --mem=100000
#
# Script name: structurethreader_K1to5_autosomal_genstr_rmRelated.sh
# Created: 2023-02-09
#
/home/scarvey/miniconda3/envs/stacks_venv/bin/structure_threader run -K 5 \
-i /mnt/data/qcarvey/stacks_workflow/07-filtered_vcfs/populations_for_geneticstructure/run3populations.snps_rmRelated.vcf.bed \
--pop option /mnt/data/qcarvey/stacks_workflow/07-filtered_vcfs/populations_for_geneticstructure/run3/popmap_plus_poporder \
-o /mnt/data/qcarvey/stacks_workflow/07-filtered_vcfs/populations_for_geneticstructure/run3/fastStructure/output \
-t 25 -fs /home/scarvey/miniconda3/envs/stacks_venv/bin/fastStructure

[StuntsPT]

Hi @quinn-ca , let me see what I can do. Can you please let me know exactly which commands you used to create your conda environment (specifically, which version of python got installed)? I'm trying to reproduce the issue here, but so far there are still a few unknowns I would like to narrow down so I can get to the bottom of the issue.

[quinn-ca]

Hi, @StuntsPT; thank you for looking into this! My python version is: Python 3.9.13. I've attached a txt file with my environment setup. Let me know if you need anything else! 20230210_stacks_venv.txt

[StuntsPT]

Ok, so here's what I've got: I renamed your file stacks_venv.yam, added a 1st line containing name: stacks and created a new conda environment using it: conda env create -f stacks_external.yml -n stacks. This did not work due to pip errors while trying to install cython, so I just removed the last 7 lines from the file (those related to pip) and re-ran the command. The final file looks like this: stacks_external.yml.txt (just remove the .txt extension that github requires). I activated the new environment, and then, used pip manually to install Structure_threader:

python3 -m pip install structure_threader

I ran the program and it works! That means a problem in your environment is unlikely. After digging a bit deeper, It might just be that you are not loading your conda environment into SLURM. Try adding this line to your sbatch file, just before you run structure_threader: conda activate stacks_venv If you still have issues, it might be some weird interaction with SLURM, and we might have to take a deeper dive. Should that be the case, please follow these instructions, and see if the problem persists:

conda create -n faststructure
conda activate faststructure
conda install python=3.9
python3 -m pip install structure_threader

Then run your sbatch script, but not before adding the line conda activate faststructure to it.

Let me know how it goes. =-)

Francisco

[quinn-ca]

Thanks for your help! I added the conda activation you suggested, and also tried adding a path to my conda.sh file (per https://github.com/conda/conda/issues/8536), but then I got a permission denied error. I tried running the script directly (not as a slurm job) but also got a permissions error. I've run many scripts via slurm and directly on the server without these permissions problems, so I'm not sure what's going on.

When I ran the code as one line (not as a script), I got the error below:

Traceback (most recent call last):
  File "/home/scarvey/miniconda3/envs/stacks_venv/lib/python3.9/site-packages/structure_threader/structure_threader.py", line 31, in <module>
    import plotter.structplot as sp
ModuleNotFoundError: No module named 'plotter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scarvey/miniconda3/envs/stacks_venv/bin/structure_threader", line 5, in <module>
    from structure_threader.structure_threader import main
  File "/home/scarvey/miniconda3/envs/stacks_venv/lib/python3.9/site-packages/structure_threader/structure_threader.py", line 41, in <module>
    import structure_threader.plotter.structplot as sp
  File "/home/scarvey/miniconda3/envs/stacks_venv/lib/python3.9/site-packages/structure_threader/plotter/structplot.py", line 22, in <module>
    import colorlover as cl
ModuleNotFoundError: No module named 'colorlover'

[StuntsPT]

If you are still having issues when running as a one line outside SLURM, than we should start there. Please follow the instructions on my previous post to create the new faststructure env, and try running structure_threader from there, without using SLURM. Once it is running independently of SLURM, we can focus on trying to run it from within SLURM.

[StuntsPT]

Closing due to inactivity. I'm assuming the issue was solved. If it wasn't, feel free to reopen the issue.