what does --file-checks do

[nsheff]

In looper run --help I see:

--file-checks Perform input file checks. Default=True.

If it defaults to True, what does turning the flag on do? And what does it do, anyway? Does it make sure 'required_files' actually exist? How would you turn it off, then? Should it be --skip-file-checks with default false?

[nsheff]

right now this is showing up on lots of commands that seems like it shouldn't... eg, table and report and check -- do we really want to be checking for input files on those, and providing a toggle to toggle it off?

I propose file checks should only be done on run (maybe runp?), and so this option should be only added to that command parser...

[nsheff]

I'm seeing this now on a routine run -- is it related to this recent change?

Traceback (most recent call last):
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 31, in __getattr__
    v = super(PathExAttMap, self).__getattribute__(item)
AttributeError: 'Project' object has no attribute 'file_checks'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/ordattmap.py", line 45, in __getitem__
    return super(OrdAttMap, self).__getitem__(item)
KeyError: 'file_checks'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 34, in __getattr__
    return self.__getitem__(item, expand)
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 51, in __getitem__
    v = super(PathExAttMap, self).__getitem__(item)
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/ordattmap.py", line 47, in __getitem__
    return AttMap.__getitem__(self, item)
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/attmap.py", line 32, in __getitem__
    return self.__dict__[item]
KeyError: 'file_checks'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nsheff/.local/bin/looper", line 10, in <module>
    sys.exit(main())
  File "/home/nsheff/.local/lib/python3.7/site-packages/looper/looper.py", line 778, in main
    run(args, rerun=(args.command == "rerun"), **compute_kwargs)
  File "/home/nsheff/.local/lib/python3.7/site-packages/looper/looper.py", line 359, in __call__
    curr_pl_fails = cndtr.add_sample(sample, rerun=rerun)
  File "/home/nsheff/.local/lib/python3.7/site-packages/looper/conductor.py", line 190, in add_sample
    if schema_source and self.prj.file_checks:
  File "/home/nsheff/.local/lib/python3.7/site-packages/looper/project.py", line 53, in __getattr__
    return getattr(self.prj, item)
  File "/home/nsheff/.local/lib/python3.7/site-packages/attmap/pathex_attmap.py", line 38, in __getattr__
    raise AttributeError(item)
AttributeError: file_checks

[stolarczyk]

yes, it's a small mistake that I made. I have it fixed locally, but want to finish sth else before I commit.

[nsheff]

oh ok thx.

[nsheff]

it worked for me locally but for some reason after upgrading on rivanna I now get this error:

[ns5bc@udc-ba36-36:~/code/glass_aml_rrbs] looper run -d -l 1
No project config defined, using: /home/nsheff/code/glass_aml_rrbs/metadata/glass_aml_rrbs_config.yaml. R
ead from dotfile (/sfs/qumulo/qhome/ns5bc/code/glass_aml_rrbs/.looper.yaml).
Traceback (most recent call last):
  File "/home/ns5bc/.local/bin/looper", line 10, in <module>
    sys.exit(main())
  File "/home/ns5bc/.local/lib/python3.6/site-packages/looper/looper.py", line 690, in main
    args = enrich_args_via_cfg(args, aux_parser)
  File "/home/ns5bc/.local/lib/python3.6/site-packages/looper/utils.py", line 211, in enrich_args_via_cfg
    elif dest in cfg_args_all:
TypeError: argument of type 'NoneType' is not iterable

[nsheff]

@stolarczyk this may be unrelated to the file checks thing, is this a separate issue? Any ideas here?

[stolarczyk]

when did you last reinstall looper and peppy? Is glass_aml_rrbs PEP any different than the one in the repo?

[nsheff]

Just now. No, it is the same. Does it work for you on Rivanna?

[stolarczyk]

I'm not able to run because the the dnameth_pipelines/piface_rrbs.yaml piface is invalid, but it gets past that point.

Worked after updating the piface

[nsheff]

are you using the .looper file? looper run -d -l 1

it fails. it works if you use looper run metadata/glass_aml_rrbs_config.yaml

Here the cfg_args_all is set to None:

https://github.com/pepkit/looper/blob/99f4d63647423b98db9988d41b268da955c27991/looper/utils.py#L203

Then here it is choking:

https://github.com/pepkit/looper/blob/99f4d63647423b98db9988d41b268da955c27991/looper/utils.py#L211

[nsheff]

I figured it out. If the .looper file points to a non-existent file, it gives this error message.

[stolarczyk]

yes, it works in both cases for me. The only extra step that I need to make is to re init the dotfile, so it reads:

Initialized looper dotfile: /sfs/qumulo/qhome/mjs5kd/code/glass_aml_rrbs/.looper.yaml

then I can do:

[mjs5kd@udc-ba36-36 glass_aml_rrbs](master): looper init metadata/glass_aml_rrbs_config.yaml 
Initialized looper dotfile: /sfs/qumulo/qhome/mjs5kd/code/glass_aml_rrbs/.looper.yaml
[mjs5kd@udc-ba36-36 glass_aml_rrbs](master): looper run -d -l 1
No project config defined, using: /sfs/qumulo/qhome/mjs5kd/code/glass_aml_rrbs/metadata/glass_aml_rrbs_config.yaml. Read from dotfile (/sfs/qumulo/qhome/mjs5kd/code/glass_aml_rrbs/.looper.yaml).
Looper version: 1.2.0-dev
Command: run
## [1 of 119] sample: AML_blast_322; pipeline: rrbs-pl
Writing script to /project/shefflab/processed/glass_aml_rrbs/submission/rrbs-pl_AML_blast_322.sub
Job script (n=1; 0.00Gb): /project/shefflab/processed//glass_aml_rrbs/submission/rrbs-pl_AML_blast_322.sub
Dry run, not submitted

Looper finished
Samples valid for job generation: 1 of 119
Commands submitted: 1 of 119
Jobs submitted: 1
Dry run. No jobs were actually submitted.
[mjs5kd@udc-ba36-36 glass_aml_rrbs](master):

[mjs5kd@udc-ba36-36 glass_aml_rrbs](master): git diff
diff --git a/.looper.yaml b/.looper.yaml
index c110ad6..7bcb4b5 100644
--- a/.looper.yaml
+++ b/.looper.yaml
@@ -1 +1 @@
-config_file_path: /home/nsheff/code/glass_aml_rrbs/metadata/glass_aml_rrbs_config.yaml
+config_file_path: /sfs/qumulo/qhome/mjs5kd/code/glass_aml_rrbs/metadata/glass_aml_rrbs_config.yaml