It would be great for ddrindex provided more useful feedback on CSV validation errors. If this error identifier the offending line and field, and printed out the offending text it would be much easier to identify problems in CSVs.
2018-09-12 16:15:16,560 DEBUG <DDR.identifier.Identifier collection:ddr-csujad-26>
2018-09-12 16:15:16,560 DEBUG Username: sara.beckman
2018-09-12 16:15:16,560 DEBUG Password: ********
2018-09-12 16:15:16,560 DEBUG idservice.IDServiceClient.login(sara.beckman)
2018-09-12 16:15:16,562 DEBUG Starting new HTTPS connection (1): idservice.densho.org:443
2018-09-12 16:15:16,759 DEBUG https://idservice.densho.org:443 "POST /api/0.1/rest-auth/login/ HTTP/1.1" 200 50
2018-09-12 16:15:16,761 INFO Checking CSV file
2018-09-12 16:15:16,780 INFO 143 rows
2018-09-12 16:15:16,780 DEBUG Guessing model based on 143 rows
2018-09-12 16:15:16,780 DEBUG model: entity
2018-09-12 16:15:16,781 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,797 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/index.json HTTP/1.1" 200 241
2018-09-12 16:15:16,817 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/language.json
2018-09-12 16:15:16,816 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/status.json
2018-09-12 16:15:16,821 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/rights.json
2018-09-12 16:15:16,825 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,825 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,835 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,836 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/facility.json
2018-09-12 16:15:16,841 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/format.json
2018-09-12 16:15:16,845 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/topics.json
2018-09-12 16:15:16,849 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,849 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,854 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/language.json HTTP/1.1" 200 1175
2018-09-12 16:15:16,854 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/status.json HTTP/1.1" 200 325
2018-09-12 16:15:16,855 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/genre.json
2018-09-12 16:15:16,860 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,860 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/rights.json HTTP/1.1" 200 597
2018-09-12 16:15:16,862 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,862 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/public.json
2018-09-12 16:15:16,867 DEBUG Starting new HTTP connection (1): partner.densho.org:80
2018-09-12 16:15:16,868 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/format.json HTTP/1.1" 200 715
2018-09-12 16:15:16,872 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/facility.json HTTP/1.1" 200 48090
2018-09-12 16:15:16,877 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 524748
2018-09-12 16:15:16,893 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/genre.json HTTP/1.1" 200 5348
2018-09-12 16:15:16,921 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/public.json HTTP/1.1" 200 302
2018-09-12 16:15:17,230 INFO Validating headers
2018-09-12 16:15:17,230 INFO ok
2018-09-12 16:15:17,230 INFO Validating rows
Traceback (most recent call last):
File "/opt/ddr-local/venv/ddrlocal/bin/ddrimport", line 14, in <module>
load_entry_point('ddr-cmdln==0.9.4b0', 'console_scripts', 'ddrimport')()
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 161, in entity
csv_path, ci, config.VOCABS_URL, idservice_client
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 286, in run_checks
chkcsv = batch.Checker.check_csv(csv_path, ci, vocabs_url)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 176, in check_csv
module, vocabs, headers, rowds, model
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 369, in _validate_csv_file
rowds_errs = csvfile.validate_rowds(module, headers, required_fields, valid_values, rowds, find_dupes)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/csvfile.py", line 206, in validate_rowds
invalid_values = find_invalid_values(module, headers, valid_values, rowds)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/csvfile.py", line 181, in find_invalid_values
bad_fields = check_row_values(module, headers, valid_values, rowd)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/csvfile.py", line 113, in check_row_values
rowd[field]
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/modules.py", line 112, in function
value = function(value)
File "/opt/ddr-local/ddr-defs/repo_models/entity.py", line 1433, in csvload_creators
def csvload_creators( text ): return converters.text_to_rolepeople(text)
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/converters.py", line 783, in text_to_rolepeople
text.split(';')
File "/opt/ddr-local/venv/ddrlocal/local/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/converters.py", line 735, in _parse_rolepeople_text
name,role = txt.split(':')
ValueError: too many values to unpack```
Tried to import entity csv for ddr-csujad-26 on local VM master 2.6.3. Got this traceback. An enhancement to traceback would be adding which line the error is located in the csv so I can more easily edit the csv file.
gjost
commented
5 years ago
It would be great for
ddrindexprovided more useful feedback on CSV validation errors. If this error identifier the offending line and field, and printed out the offending text it would be much easier to identify problems in CSVs.