search

wtsi-hgi / metadata-check

Tests to perform on the iRODS metadata for a BAM file, versus the information in the file header
2 stars 3 forks source link

Impossible to run the checks for large studies when fetching loads of data from iRODS #11

Closed irinaColgiu closed 8 years ago

irinaColgiu commented 8 years ago

It seems that when trying to run the checks with the option of fetch_by_metadata on large studies (fetching loads of metadata from iRODS), I get the following error:

Traceback (most recent call last): File "mcheck/main/run_checks.py", line 174, in main() File "mcheck/main/run_checks.py", line 157, in main study_internal_id, fpaths_irods, irods_zone) File "mcheck/main/run_checks.py", line 92, in check_metadata irods_metadata_dict = MetadataSelfChecks.fetch_and_preprocess_irods_metadata_by_metadata(search_criteria, irods_zone, issues_dict, reference) File "/nfs/users/nfs_i/ic4/Projects/python3/meta-check/mcheck/checks/mchecks_by_type.py", line 48, in fetch_and_preprocess_irods_metadata_by_metadata irods_zone) File "/nfs/users/nfs_i/ic4/Projects/python3/meta-check/mcheck/metadata/irods_metadata/irods_meta_provider.py", line 59, in retrieve_raw_files_metadata_by_metadata list_of_data_objs_and_metadata = connection.data_object.get_by_metadata(search_crit_list, zone=zone) File "/nfs/users/nfs_i/ic4/Projects/python3/meta-check/ENV/lib/python3.5/site-packages/baton/_baton/baton_entity_mappers.py", line 77, in get_by_metadata baton_out_as_json = self.run_baton_query(BatonBinary.BATON_METAQUERY, arguments, input_data=baton_json) File "/nfs/users/nfs_i/ic4/Projects/python3/meta-check/ENV/lib/python3.5/site-packages/baton/_baton/_baton_runner.py", line 68, in run_baton_query baton_out = self._run_command(program_arguments, input_data=input_data) File "/nfs/users/nfs_i/ic4/Projects/python3/meta-check/ENV/lib/python3.5/site-packages/baton/_baton/_baton_runner.py", line 108, in _run_command out, error = process.communicate(input=input_data, timeout=timeout_in_seconds) File "/nfs/users/nfs_i/ic4/Projects/Python-3.5.0/.localpython/lib/python3.5/subprocess.py", line 1068, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/nfs/users/nfs_i/ic4/Projects/Python-3.5.0/.localpython/lib/python3.5/subprocess.py", line 1698, in _communicate ready = selector.select(timeout) File "/nfs/users/nfs_i/ic4/Projects/Python-3.5.0/.localpython/lib/python3.5/selectors.py", line 367, in select fd_event_list = self._poll.poll(timeout) KeyboardInterrupt

colin-nolan commented 8 years ago

The exception suggests that SIGINT was given mid-query - are sure it wasn't killed? Can the same thing be reproduced?

irinaColgiu commented 8 years ago

Yes, it is reproducible for any long study (e.g. ddd, interval, helic). It's not being killed.

colin-nolan commented 8 years ago

If you can provide the call to the baton wrapper, I shall investigate the issue further.

colin-nolan commented 8 years ago

Testing with baton wrapper to see if the issue occurs outside of metadata check.

irinaColgiu commented 8 years ago

My suspicion is that it is not baton wrapper, but metacheck, the way it processes metadata when the size is large.

colin-nolan commented 8 years ago

I didn't encouter the problem when the baton wrapper was used in isolation; as you suspected, this looks like it's an issue with metacheck.

If you want the JSON for Pomak (to save waiting an age fetching it again from iRODS), I saved it to ~/shared/temp/IHTP_MWGS_HELICPomak.json (my home).

irinaColgiu commented 8 years ago

It seems that this issue was due to insufficient memory when the baton was running as a subprocess.