dgordon562 commented 1 month ago

Could I be doing something wrong? Or has something changed so this no longer works? Or is pytest dependent on something you have installed at your lab that you were unaware of?

conda activate cenmap_dev (cenmap_dev) {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw pytest -vvv ======================================= test session starts ======================================== platform linux -- Python 3.12.0, pytest-8.3.3, pluggy-1.5.0 -- /home/hsiehph/shared/conda_shared/envs/cenmap_dev/bin/python cachedir: .pytest_cache rootdir: /panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw collected 9 items / 2 errors

============================================== ERRORS ============================================== _ ERROR collecting workflow/rules/CDR-Finder/test/calculate_windows/test_calculate_windows.py __ ImportError while importing test module '/panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw/workflow/rules/CDR-Finder/test/calculate_windows/test_calculate_windows.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/_pytest/python.py:493: in importtestmodule mod = import_path( /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/_pytest/pathlib.py:582: in import_path importlib.import_module(module_name) /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/importlib/init.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level)

:1381: in _gcd_import ??? :1354: in _find_and_load ??? :1304: in _find_and_load_unlocked ??? :488: in _call_with_frames_removed ??? :1381: in _gcd_import ??? :1354: in _find_and_load ??? :1318: in _find_and_load_unlocked ??? E ModuleNotFoundError: No module named 'test.calculate_windows' _________________ ERROR collecting workflow/rules/CDR-Finder/test/cdr/test_cdr.py __________________ ImportError while importing test module '/panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw/workflow/rules/CDR-Finder/test/cdr/test_cdr.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/_pytest/python.py:493: in importtestmodule mod = import_path( /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/_pytest/pathlib.py:582: in import_path importlib.import_module(module_name) /home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) :1381: in _gcd_import ??? :1354: in _find_and_load ??? :1304: in _find_and_load_unlocked ??? :488: in _call_with_frames_removed ??? :1381: in _gcd_import ??? :1354: in _find_and_load ??? :1318: in _find_and_load_unlocked ??? E ModuleNotFoundError: No module named 'test.cdr' ===================================== short test summary info ====================================== ERROR workflow/rules/CDR-Finder/test/calculate_windows/test_calculate_windows.py ERROR workflow/rules/CDR-Finder/test/cdr/test_cdr.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================== 2 errors in 0.36s ========================================= (cenmap_dev) {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw >

koisland commented 1 month ago

Hi. This is because pytest attempts to run tests in the workflow/rules/CDR-Finder submodule. I have updated the wiki documentation to change this. The command will be.

pytest test/ -vvv

dgordon562 commented 4 weeks ago

Thanks, Keith! That got me further:

conda activate cenmap_dev (cenmap_dev) {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw pytest test/ -vvv

This gives these errors (below) plus another 30 pages of error messages. I'm happy to supply any additional information you would like.

Best wishes, David (give my regards to Glennis) ======================================= test session starts ======================================== platform linux -- Python 3.12.0, pytest-8.3.3, pluggy-1.5.0 -- /home/hsiehph/shared/conda_shared/envs/cenmap_dev/bin/python cachedir: .pytest_cache rootdir: /panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw collected 9 items

test/count_cens/test_count_cens.py::test_count_cens[test/count_cens/all_AS-HOR_lengths.tsv-test/count_cens/expected_counts.tsv] FAILED [ 11%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19129_chr16_hap2_forward.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/NA19129_chr16_hap2_forward_filtered.bed] FAILED [ 22%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19317_chr16_hap2_reverse.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/NA19317_chr16_hap2_reverse_filtered.bed] FAILED [ 33%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19317_chr3_hap2_forward.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr3-test/filter_dnabrnn_output/expected/NA19317_chr3_hap2_forward_filtered.bed] FAILED [ 44%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/HG03248_chr16_hap1_reverse.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/HG03248_chr16_hap1_reverse_filtered.bed] FAILED [ 55%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/HG00512_cens.bed-test/map_cens/expected/HG00512_expected_cens.bed] FAILED [ 66%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/NA19705_cens.bed-test/map_cens/expected/NA19705_expected_cens.bed] FAILED [ 77%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/NA20355_cens.bed-test/map_cens/expected/NA20355_expected_cens.bed] FAILED [ 88%] test/rename_rm/test_rename_rm.py::test_rename_rm[test/rename_rm/HG01596_renamed.fa.out-test/rename_rm/HG01596.fa.fai-test/rename_rm/HG01596_renamed.fa.fai-test/rename_rm/HG01596_expected.fa.out] FAILED [100%]

_ test_count_cens[test/count_cens/all_AS-HOR_lengths.tsv-test/count_cens/expected_counts.tsv] __

infile = 'test/count_cens/all_AS-HOR_lengths.tsv', expected = 'test/count_cens/expected_counts.tsv'

@pytest.mark.parametrize(
    ["infile", "expected"],
    [
        (
            "test/count_cens/all_AS-HOR_lengths.tsv",
            "test/count_cens/expected_counts.tsv",
        )
    ],
)
def test_count_cens(infile: str, expected: str):

  run_integration_test(
"python", "workflow/scripts/count_complete_cens.py", "-i", infile, expected_output=expected, )

test/count_cens/test_count_cens.py:15:

test/helpers/integration.py:5: in run_integration_test process = subprocess.run(

input = None, capture_output = True, timeout = None, check = True popenargs = (['python', 'workflow/scripts/count_complete_cens.py', '-i', 'test/count_cens/all_AS-HOR_lengths.tsv'],) kwargs = {'stderr': -1, 'stdout': -1} process = <Popen: returncode: 1 args: ['python', 'workflow/scripts/count_complete_cens...> stdout = b'' stderr = b'Traceback (most recent call last):\n File "/panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw/workflow/scripts/count_complete_cens.py", line 70, in \n raise SystemExit(main())\n ^^^^^^\n File "/panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw/workflow/scripts/count_complete_cens.py", line 33, in main\n pl.read_csv(\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 92, in wrapper\n return function(*args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 92, in wrapper\n return function(*args, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 92, in wrapper\n return function(args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/io/csv/functions.py", line 536, in read_csv\n return _update_columns(df, new_columns)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/io/csv/_utils.py", line 37, in _update_columns\n df.columns = list(new_columns)\n ^^^^^^^^^^\n File "/home/hsiehph/shared/conda_shared/envs/cenmap_dev/lib/python3.12/site-packages/polars/dataframe/frame.py", line 842, in columns\n self._df.set_column_names(names)\npolars.exceptions.ShapeError: 4 column names provided for a DataFrame of width 1\n' retcode = 1 . . .

dgordon562 commented 4 weeks ago

Thanks, Keith! I tried that and got further but not successful. Please see the details on github. I'm happy to work with you and supply you whatever you would like.

On Thu, Oct 17, 2024 at 6:10 AM Keith Oshima @.***> wrote:

Hi. This is because pytest attempts to run tests in the workflow/rules/CDR-Finder submodule. I have updated the wiki documentation to change this. The command will be.

pytest test/ -vvv

— Reply to this email directly, view it on GitHub https://github.com/logsdon-lab/CenMAP/issues/89#issuecomment-2419506513, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX6XC2US7UMSMGDLS2GQ6DZ36ZL3AVCNFSM6AAAAABQCNDUQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJZGUYDMNJRGM . You are receiving this because you authored the thread.Message ID: @.***>

koisland commented 3 weeks ago

Hi @dgordon562,

The fact that all the tests failed due to empty data means that Git LFS didn't pull the test files, and Polars was reading the pointer file.

I have it installed by default so I overlooked it. I updated the wiki page and env_dev.yaml in PR #95 so future users won't get the same issue.

Try this and let me know if this resolves your issue.

conda install git-lfs
git lfs pull
pytest test/ -vvv

Cheers, Keith

dgordon562 commented 3 weeks ago

thanks, Keith!!! (could you get me your email address so I don't have to ask you embarrassing, stupid questions in public? you could send it to dgordon562@gmail.com)

dgordon562 commented 3 weeks ago

hmmm.... in the wiki you now say:

Setup environments.

conda env create --name cenmap_dev -f env_dev.yaml conda activate cenmap_dev

Pull test data.

git lfs pull

but "git lfs pull" returns git: 'lfs' is not a git command. See 'git --help'.

so I tried: git pull lfs which returned: fatal: 'lfs' does not appear to be a git repository

I assume there is some typo that I should have been able to figure out...?

dgordon562 commented 3 weeks ago

git pull You are not currently on a branch. Please specify which branch you want to merge with. See git-pull(1) for details.

git pull <remote> <branch>

(cenmap_dev2) {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP

git pull origin master fatal: couldn't find remote ref master

dgordon562 commented 3 weeks ago

finally got lfs installed.

export PATH=/home/hsiehph/gordo893/packages/git-lfs/git-lfs-3.5.1/installed/bin:$PATH {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP git lfs pull {gordo893}acn84:/home/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP

But then pytest failed again:

pytest test/ -vvv ======================================= test session starts ======================================== platform linux -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0 -- /common/software/install/migrated/anaconda/anaconda3-5.0.1/bin/python cachedir: .cache rootdir: /panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP, inifile: plugins: hypothesis-3.38.5 collected 9 items

test/count_cens/test_count_cens.py::test_count_cens[test/count_cens/all_AS-HOR_lengths.tsv-test/count_cens/expected_counts.tsv] FAILED [ 11%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19129_chr16_hap2_forward.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/NA19129_chr16_hap2_forward_filtered.bed] FAILED [ 22%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19317_chr16_hap2_reverse.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/NA19317_chr16_hap2_reverse_filtered.bed] FAILED [ 33%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19317_chr3_hap2_forward.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr3-test/filter_dnabrnn_output/expected/NA19317_chr3_hap2_forward_filtered.bed] FAILED [ 44%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/HG03248_chr16_hap1_reverse.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/HG03248_chr16_hap1_reverse_filtered.bed] FAILED [ 55%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/HG00512_cens.bed-test/map_cens/expected/HG00512_expected_cens.bed] FAILED [ 66%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/NA19705_cens.bed-test/map_cens/expected/NA19705_expected_cens.bed] FAILED [ 77%] test/map_cens/test_map_cens.py::test_map_cens[test/map_cens/input/NA20355_cens.bed-test/map_cens/expected/NA20355_expected_cens.bed] FAILED [ 88%] test/rename_rm/test_rename_rm.py::test_rename_rm[test/rename_rm/HG01596_renamed.fa.out-test/rename_rm/HG01596.fa.fai-test/rename_rm/HG01596_renamed.fa.fai-test/rename_rm/HG01596_expected.fa.out] FAILED [100%]

koisland commented 3 weeks ago

Hi @dgordon562. I prefer to keep issues public to help future users. I missed adding instructions for installing the Git LFS hooks in the repo. This should work.

git lfs install
git lfs pull
pytest test/ -vvv

dgordon562 commented 3 weeks ago

Hi, Keith, I think you missed my latest comment. I did install lfs but pytest still gave errors. I did exactly the commands you specified here except that I prefaced them with export PATH=/home/hsiehph/gordo893/packages/git-lfs/git-lfs-3.5.1/installed/bin:$PATH If I don't do that first, then the git lfs commands fail with: git: 'lfs' is not a git command. See 'git --help' But I lfs commands are now working for me. pytest is not.
Do you need more information from me to troubleshoot pytest? Thanks! David

koisland commented 3 weeks ago

Hi @dgordon562 ,

Can you see what test/filter_dnabrnn_output/input/HG03248_chr16_hap1_reverse.bed looks like?

head test/filter_dnabrnn_output/input/HG03248_chr16_hap1_reverse.bed

I can reproducibly run all tests in a container so I still think you're missing a step with lfs. That is the only explanation I see for why all tests would fail.

docker run -it --rm condaforge/mambaforge

git clone https://github.com/logsdon-lab/CenMAP.git && cd CenMAP
# Remove snakemake due to datrie install issues
grep -v "snakemake"  env_dev.yaml > end_dev_no_smk.yaml
conda env create -f end_dev_no_smk.yaml --name cenmap && conda activate cenmap
git lfs install && git lfs pull
pytest test/ -vvv

dgordon562 commented 3 weeks ago

Thanks for looking at this, Keith. Here is what you asked for (below). What does this tell you?

head test/filter_dnabrnn_output/input/HG03248_chr16_hap1_reverse.bed HG03248_chr16_haplotype1-0000015:120609-26564571 1318250 1318306 2 HG03248_chr16_haplotype1-0000015:120609-26564571 13717950 13718100 2 HG03248_chr16_haplotype1-0000015:120609-26564571 14412650 14412702 2 HG03248_chr16_haplotype1-0000015:120609-26564571 16961850 16962076 2 HG03248_chr16_haplotype1-0000015:120609-26564571 18324300 18324400 2 HG03248_chr16_haplotype1-0000015:120609-26564571 24870350 24870442 2 HG03248_chr16_haplotype1-0000015:120609-26564571 25731850 25731950 2 {gordo893}cn1054:/home/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP

koisland commented 3 weeks ago

Okay. That's useful and means that LFS isn't the problem.

Looking more closely at your previous logs, the wrong python version (3.6.3) is being used with pytest. Can you rerun the test with python==3.12?

Your PATH might be overriding the python version in the cenmap conda environment you created with env_dev.yaml

pytest test/ -vvv
======================================= test session starts ========================================
platform linux -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0 -- /common/software/install/migrated/anaconda/anaconda3-5.0.1/bin/python
cachedir: .cache
rootdir: /panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw2/CenMAP, inifile:
plugins: hypothesis-3.38.5
collected 9 items

dgordon562 commented 3 weeks ago

fails same as before. Even with a clean git clone.

To be absolutely clear, this is what I did:

conda activate cenmap_dev for lfs, I must: export PATH=/home/hsiehph/gordo893/packages/git-lfs/git-lfs-3.5.1/installed/bin:$PATH (If I don't, git lfs will results in "lfs not a git command") mkdir raw3 (cenmap_dev) {gordo893}cn1054:/home/hsiehph/shared/software/pipelines/glennis

cd raw3 (cenmap_dev) {gordo893}cn1054:/home/hsiehph/shared/software/pipelines/glennis/raw3 git clone git@github.com:logsdon-lab/CenMAP.git --branch v0.2.2 --recurse-submodules cd CenMAP (cenmap_dev) {gordo893}cn1054:/home/hsiehph/shared/software/pipelines/glennis/raw3/CenMAP pytest test/ -vvv ======================================= test session starts ======================================== platform linux -- Python 3.12.0, pytest-8.3.3, pluggy-1.5.0 -- /home/hsiehph/shared/conda_shared/envs/cenmap_dev/bin/python cachedir: .pytest_cache rootdir: /panfs/jay/groups/7/hsiehph/shared/software/pipelines/glennis/raw3/CenMAP collected 9 items

test/count_cens/test_count_cens.py::test_count_cens[test/count_cens/all_AS-HOR_lengths.tsv-test/count_cens/expected_counts.tsv] FAILED [ 11%] test/filter_dnabrnn_output/test_filter_dnabrnn_output.py::test_filter_dnabrnn_output[test/filter_dnabrnn_output/input/NA19129_chr16_hap2_forward.bed-test/filter_dnabrnn_output/input/dnabrnn_thresholds.json-chr16-test/filter_dnabrnn_output/expected/NA19129_chr16_hap2_forward_filtered.bed] FAILED [ 22%] . . . (more errors)

Perhaps I could modify test_count_cens.py to show more debugging information?

dgordon562 commented 3 weeks ago

also did git lfs pull and still failed. checked python: which python /home/hsiehph/shared/conda_shared/envs/cenmap_dev/bin/python (cenmap_dev) {gordo893}cn1054:/home/hsiehph/shared/software/pipelines/glennis/raw3/CenMAP

python --version Python 3.12.0

logsdon-lab / CenMAP

github test immediately fails #89

Setup environments.

Pull test data.