Closed Alex-Karmazin closed 2 years ago
Also a got such traceback in console during this error: """ Running gene mapper... multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\multiprocessing\pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\base\mp_runners.py", line 73, in mapper_runner output = genemapper.run_as_slave(pos_no) File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\base\mapper.py", line 316, in run_as_slave for ln, line, crv_data in self.reader.loop_data(): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\inout.py", line 146, in loop_data for lnum, toks in self._loop_data(): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\inout.py", line 212, in _loop_data csv.field_size_limit(sys.maxsize) OverflowError: Python int too large to convert to C long """
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\Scripts\oc.exe__main.py", line 7, in
@Alex-Karmazin Thanks. I'll look into it. The BadFormatError
message is a misleading message which was suppressed by OpenCRAVAT. The overflow error was caused by the combination of a fix to handle very long annotation by some modules (thus, current OpenCRAVAT will err with such input and modules) and Python's using different maximum integer value in different systems. Both have been corrected in oakvar 2.5.11 I have just released. Please try that one. Also, vcf-converter 2.1.6 should be installed. Below are the steps.
pip install -U oakvar
ov store fetch
ov module update vcf-converter
here is the input file that caused that issue https://drive.google.com/file/d/1ebb1-opKU4F6UFLUsxyQnmjVCgtWeiYd/view?usp=sharing
@antonkulaga Thanks. I have confirmed with your input file, oakvar v2.5.12, and vcf-converter 2.1.6 that the reported errors do not occur anymore. Let me know if it works for you as well.
I got many such errors: """ Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\websubmit\websubmit.py", line 687, in get_jobs job = await get_job(request, job_id) File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\websubmit\websubmit.py", line 662, in get_job LooseVersion(job.info["open_cravat_version"]) File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\distutils\version.py", line 52, in lt c = self._cmp(other) File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\distutils\version.py", line 339, in _cmp if self.version == other.version: AttributeError: 'LooseVersion' object has no attribute 'version' """ But they not interrupt process, so it is not critical for now. Thank you for quick support.
@Alex-Karmazin What is the Python version in your environment?
python 3.9
@Alex-Karmazin Thanks. Python 3.9 seems fine for using LooseVersion, but anyway LooseVersion is being deprecated in Python anyway, so I replaced it with packaging.version
which is recommended by Python. The latest oakvar should not have that error.
I got some new tracebacks:
"""
Running annotators...
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 297, in load_class
module = import_module(module_name)
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\importlib__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 307, in load_class
loader.exec_module(module)
File "
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 307, in load_class
loader.exec_module(module)
File "
It seems my module that I'm working on "prs" did not load properly. But on opencravat it was working well.
@Alex-Karmazin I'll take a look.
@Alex-Karmazin Can you do ov system md
and see if the output is a directory which has your psr
module under annotators
?
Yes it points to valid path. Here is this module if you need it. https://drive.google.com/file/d/1bKV_7AWKktXWB_aRtEt18UGyUSvchtA4/view?usp=sharing
On my system, prs
module was successfully loaded and did not produce the module loading error. Just division by zero
error was produced. What is the version obtained with ov version
? The latest is 2.5.12.
@Alex-Karmazin Regarding the error with dbsnp
, I have just published a new version 154.0.2.1
with a fix.
Now after I managed to update dbsnp I could come back to this issue. I have strange error:
"""
2022/08/10 20:38:56 prs module loading error for prs
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 297, in load_class
module = import_module(module_name)
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\importlib__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 306, in load_class
loader.exec_module(module)
File "
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\base\mp_runners.py", line 31, in annot_from_queue annotator_class = load_class(module.script_path, "Annotator") File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\util\util.py", line 308, in load_class raise ModuleLoadingError(module_name) oakvar.exceptions.ModuleLoadingError: module loading error for prs """ It is the same custom module for PRS but I did not know why it searching file for this path 'C:\dev\python\openCravatPlugin\prs\prs.py'. It may be cached somehow. There is no such path inside the code, and it was long ago when this path had any sense. Command "ov system md" has a valid path "D:\dev\oakVar\modules". Where I could find any clue why it search module in the old path?
Here is little bit more information: """ 2022/08/10 20:38:50 oakvar C:\ProgramData\Anaconda3\envs\openCravatPlugin\Scripts\oc run C:\open-cravat\jobs\default\220810-203847\longevity_snps_1000genom.vcf -a dbsnp prs -l hg38 --skip reporter --temp-files 2022/08/10 20:38:50 oakvar started: Wed Aug 10 20:38:49 2022 2022/08/10 20:38:50 oakvar conf file: None 2022/08/10 20:38:50 oakvar input files: C:\open-cravat\jobs\default\220810-203847\longevity_snps_1000genom.vcf 2022/08/10 20:38:50 oakvar input assembly: hg38 2022/08/10 20:38:50 oakvar version: oakvar 2.5.23 C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar 2022/08/10 20:38:50 oakvar version: dbsnp 154.0.2.1 D:\dev\oakVar\modules\annotators\dbsnp 2022/08/10 20:38:50 oakvar version: prs None C:\dev\python\openCravatPlugin\prs 2022/08/10 20:38:50 oakvar version: hg38 1.10.2 D:\dev\oakVar\modules\mappers\hg38 2022/08/10 20:38:50 oakvar.converter started: Wed Aug 10 20:38:50 2022 2022/08/10 20:38:50 oakvar.converter Input file(s): C:\open-cravat\jobs\default\220810-203847\longevity_snps_1000genom.vcf 2022/08/10 20:38:50 oakvar.converter version: vcf-converter==2.1.6 2022/08/10 20:38:50 oakvar.converter input format: vcf 2022/08/10 20:38:51 oakvar.converter error lines: 0 2022/08/10 20:38:51 oakvar.converter finished: Wed Aug 10 20:38:51 2022 2022/08/10 20:38:51 oakvar.converter num input lines: 2797 2022/08/10 20:38:51 oakvar.converter runtime: 0.303 2022/08/10 20:38:51 oakvar num_workers: 7 2022/08/10 20:38:51 oakvar input line chunksize=624 total number of input lines=4369 number of chunks=7 2022/08/10 20:38:55 oakvar num_workers: 7 """
So far this issue is a blocker for our PRS module
ov run
can accept the path to a module as an option to -a
argument. For example, ov run input.vcf -a /path/to/prs
. This way, testing and using custom modules outside of the system modules directory is easy. -a
as a custom path has priority over -a
as a module name.
Thus, if there is prs
directory in the current working directory, the error can happen.
If there is no more prs
in the current working directory and the error still occurs, this can happen if ov gui
has been running while the custom prs
directory was removed.
To test these possibilities:
ov run ... -a prs
command in a directory in which there is no prs
directoryov gui
in a directory in which there is no prs
directoryDoes any of the above make sense in your case?
Thank you, it helped. There was prs folder in my project where I put everything regarding this functionality. I just rename it and everything starts working. By the way in OakVar secondary_data structure is changed compared with OpenCravat. Do you plan to have backward compatibility, or it will be incompatible with it?
@Alex-Karmazin I plan to keep backward compatibility and thought that it has been kept. Can you describe the discrepancy a bit so that I can start investigation?
In OpenCravat I write secondary_data['dbsnp'][0]['rsid'] but now I have to change it to secondary_data['dbsnp']['rsid'] in order to get it working.
@Alex-Karmazin Ah, got it. I pushed a new release oakvar 2.5.27 which restored the backward compatibility. Please try that one with the old format for secondary_data.
Unfortunatly new errors appeared:
"""
No module named 'vcf'
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\cruise_misle\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\ProgramData\Anaconda3\envs\cruise_misle\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\envs\cruise_misle\Scripts\oc.exe__main.py", line 7, in
After modules update, these errors disappears. But still it is incompatible with opencravat secondary_data['dbsnp'][0]['rsid'] is not working with [0] only secondary_data['dbsnp']['rsid'] works.
I see. Have you updated OakVar with "pip install -U oakvar" as well?
Yes now I have Version: 2.5.23
@Alex-Karmazin just don't be lazy with updates, I am using 2.5.27 already
I updated to Version: 2.5.27 and now it is compatible with opencravat.
I start job with file that was succesfuly processed with opencravat. But with oak var 2.5.10 it get folowing error: """ 2022/08/01 20:53:11 oakvar C:\ProgramData\Anaconda3\envs\openCravatPlugin\Scripts\oc run C:\open-cravat\jobs\default\220801-205305\antonkulaga_38_all_rsid_chrompos.vcf -a dbsnp prs -l hg38 --skip reporter --temp-files 2022/08/01 20:53:11 oakvar started: Mon Aug 1 20:53:11 2022 2022/08/01 20:53:11 oakvar conf file: None 2022/08/01 20:53:12 oakvar input files: C:\open-cravat\jobs\default\220801-205305\antonkulaga_38_all_rsid_chrompos.vcf 2022/08/01 20:53:12 oakvar input assembly: hg38 2022/08/01 20:53:12 oakvar version: oakvar 2.5.10 C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar 2022/08/01 20:53:12 oakvar version: dbsnp 154.0.2 C:\dev\python\openCravatPlugin\modules\annotators\dbsnp 2022/08/01 20:53:12 oakvar version: prs None prs 2022/08/01 20:53:12 oakvar version: hg38 1.10.2 C:\dev\python\openCravatPlugin\modules\mappers\hg38 2022/08/01 20:53:12 oakvar.converter started: Mon Aug 1 20:53:12 2022 2022/08/01 20:53:12 oakvar.converter Input file(s): C:\open-cravat\jobs\default\220801-205305\antonkulaga_38_all_rsid_chrompos.vcf 2022/08/01 20:53:17 oakvar.converter input format: vcf 2022/08/01 20:53:17 oakvar.converter Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\openCravatPlugin\lib\site-packages\oakvar\base\converter.py", line 26, in convert_file yield ln, line, self.convert_line(line) File "C:\dev\python\openCravatPlugin\modules\converters\vcf-converter\vcf-converter.py", line 216, in convert_line raise BadFormatError('All samples have the reference genotype.') oakvar.exceptions.BadFormatError: All samples have the reference genotype. 2022/08/01 21:07:51 oakvar.converter error lines: 1418835 2022/08/01 21:07:51 oakvar.converter finished: Mon Aug 1 21:07:51 2022 2022/08/01 21:07:51 oakvar.converter num input lines: 4660909 2022/08/01 21:07:51 oakvar.converter runtime: 874.169 2022/08/01 21:07:52 oakvar num_workers: 7 2022/08/01 21:07:57 oakvar input line chunksize=675344 total number of input lines=4727413 number of chunks=7 """