o2r-project / o2r-meta

Metadata toolsuite for an extract-map-validate workflow supporting reproducible research
Apache License 2.0
2 stars 3 forks source link

Loader Error caused by RData-Parser #87

Closed Timmimim closed 6 years ago

Timmimim commented 6 years ago

Loader fails to upload repositories holding an .RData file.

loader_1          | 2018-02-13T16:19:10.462Z loader [wC06W] [extract container] [o2rmeta] 20180213.161910 received arguments: {'debug': True, 'tool': 'extract', 'formats': False, 'inputdir': '/tmp/o2r/compendium/wC06W', 'outputdir': '/tmp/o2r/compendium/wC06W/.erc', 'outputtostdout': False, 'ercid': 'wC06W', 'basedir': '/tmp/o2r/compendium/wC06W', 'modexml': False, 'stayoffline': False, 'metafiles': True}
loader_1          | [o2rmeta] 20180213.161910 launching extractor
loader_1          | 2018-02-13T16:19:11.665Z loader [wC06W] [extract container] [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_bagittxt.ParseBagitTxt object at 0x7f8e2047af28>
loader_1          | [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_displayfiles.ParseDisplayFiles object at 0x7f8e2047af60>
loader_1          | [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_geojson.ParseGeojson object at 0x7f8e2047aa20>
loader_1          | [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_netcdf.ParseNetcdf object at 0x7f8e2047a470>
loader_1          | 2018-02-13T16:19:11.665Z loader [wC06W] [extract container] [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_ogc_shp.ParseGeopackage object at 0x7f8e2047a668>
loader_1          | 2018-02-13T16:19:11.666Z loader [wC06W] [extract container] [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_rmd.ParseRmd object at 0x7f8e20484128>
loader_1          | [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_rdata.ParseRData object at 0x7f8e20484b38>
loader_1          | [o2rmeta][debug: extract.metaextract @ register_parsers] 20180213.161911 <parsers.parse_yaml.ParseYaml object at 0x7f8e20438358>
loader_1          | 2018-02-13T16:19:11.668Z loader [wC06W] [extract container] [o2rmeta][debug: parsers.parse_rdata @ parse] 20180213.161911 dictionary update sequence element #0 has length 1; 2 is required
loader_1          | Traceback (most recent call last):
loader_1          |   File "/o2r-meta/o2rmeta.py", line 78, in <module>
loader_1          | 2018-02-13T16:19:11.669Z loader [wC06W] [extract container] metaextract.start(dbg=argsd['debug'], i=argsd['inputdir'], o=argsd['outputdir'], s=argsd['outputtostdout'], xo=argsd['stayoffline'], e=argsd['ercid'], f=argsd['formats'], b=argsd['basedir'], m=argsd['metafiles'], xml=argsd['modexml'])
loader_1          |   File "/o2r-meta/extract/metaextract.py", line 330, in start
loader_1          | 2018-02-13T16:19:11.669Z loader [wC06W] [extract container] cache_extracted = x.parse(p=full_file_path, ext=file_extension, of=output_format, om=output_mode, md=MASTER_MD_DICT, bd=basedir, m=True, is_debug=is_debug, xo=stay_offline)
loader_1          |   File "/o2r-meta/parsers/parse_rdata.py", line 59, in parse
loader_1          | 2018-02-13T16:19:11.670Z loader [wC06W] [extract container] rdata_dict[path_file].update(key)
loader_1          | ValueError: dictionary update sequence element #0 has length 1; 2 is required

@nuest fyi

nuest commented 6 years ago

It would also be feasible to simply unregister the rdata-parser, although I am not sure about the consequences... probably rdata-files are not included in the data file listing then anymore.

ghost commented 6 years ago

ok, PR is there, I tried to fix this error which puzzles me a little: for some reason the filename would have only one char and for some reason the dictionary method could not handle it. To be save I also made the function return the MD if rdata parsing fails for any reason.