search

denshoproject / ddr-cmdln

Command-line tools for automating the Densho Digital Repository's various processes.
Other
0 stars 2 forks source link

ddrimport entity throws error: 'unicode' object has no attribute 'path_abs' #152

Closed pkikawa closed 5 years ago

pkikawa commented 5 years ago

284 master

ddrimport entity error. sara sanity checked me to make sure i wasn't especially insane. 

$ ddrimport entity -U philip.kikawa -P REDACTED /media/sf_ddrshared/test\ csv/ddr-testing-40226-objects.csv /media/qnfs/kinkura/gold/ddr-testing-40226 | tee -a /media/sf_ddrshared/test\ csv/ddr-testing-40226ObjImp.log
Traceback (most recent call last):
  File "/opt/ddr-local/venv/ddrlocal/bin/ddrimport", line 14, in <module>
    load_entry_point('ddr-cmdln==0.9.4b0', 'console_scripts', 'ddrimport')()
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 161, in entity
    csv_path, ci, config.VOCABS_URL, idservice_client
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 291, in run_checks
    chkeids = batch.Checker.check_eids(chkcsv['rowds'], ci, idservice_client)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 265, in check_eids
    rowds, cidentifier.model, cidentifier.path_abs()
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 348, in _ids_in_local_repo
    already = [i for i in new_ids if i in existing_ids]
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/identifier.py", line 1070, in __eq__
    return self.path_abs() == other.path_abs()
AttributeError: 'unicode' object has no attribute 'path_abs'

maybe we're both insane

EDIT (gjost): Formatting for traceback. EDIT (gjost): Pasted command before traceback. EDIT (gjost): Redacted password in command.

pkikawa commented 5 years ago

anyway, this was running ddrimport entity -U philip.kikawa -P REDACTED /media/sf_ddrshared/test\ csv/ddr-testing-40226-objects.csv /media/qnfs/kinkura/gold/ddr-testing-40226 | tee -a /media/sf_ddrshared/test\ csv/ddr-testing-40226ObjImp.log

sara ran the same thing and got the same error.

EDIT (gjost): Redacted password.

gjost commented 5 years ago

Can you include a copy of the CSV?

pkikawa commented 5 years ago

ddr-testing-40226-objects.zip

gjost commented 5 years ago

Thanks, using it for testing.

On May 15, 2019 12:38:26 PM PDT, Philip Kikawa notifications@github.com wrote:

ddr-testing-40226-objects.zip

-- You are receiving this because you were assigned. Reply to this email directly or view it on GitHub: https://github.com/densho/ddr-cmdln/issues/152#issuecomment-492795022

-- "Work as if you live in the early days of a better national. --Alasdair Gray

gjost commented 5 years ago

This was a data problem. The CSV had a number of problems:

Here's my edited CSV file (I changed all the IDs to avoid ID collisions): ddr-testing-40151-objects.zip You can view the differences in a diff tool. I recommend Meld (http://meldmerge.org/) which works on Linux and Windows.

The Python CSV library can read a lot of things but it can't always read your mind. DDR uses the following settings when exporting to CSV.

You'll get better results if you use the same settings when exporting spreadsheet docs to CSV before you try to import.

pkikawa commented 5 years ago

i used your edited CSV and encountered the exact same errror

(ddrlocal) ddr@DDREditor:/media/qnfs/kinkura/gold/ddr-testing-40227$ ddrimport entity -U philip.kikawa -P Qk325rrMx /media/sf_ddrshared/ddr-testing-40227-obj.csv /media/qnfs/kinkura/gold/ddr-testing-4
0227
2019-05-16 11:14:42,804 DEBUG    <DDR.identifier.Identifier collection:ddr-testing-40227>
2019-05-16 11:14:42,804 DEBUG    Username: philip.kikawa
2019-05-16 11:14:42,804 DEBUG    Password: *********
2019-05-16 11:14:42,805 DEBUG    idservice.IDServiceClient.login(philip.kikawa)
2019-05-16 11:14:42,808 DEBUG    Starting new HTTPS connection (1): idservice.densho.org
2019-05-16 11:14:42,981 DEBUG    https://idservice.densho.org:443 "POST /api/0.1/rest-auth/login/ HTTP/1.1" 200 50
2019-05-16 11:14:42,984 INFO     Checking CSV file
2019-05-16 11:14:42,992 INFO     22 rows
2019-05-16 11:14:42,992 DEBUG    Guessing model based on 22 rows
2019-05-16 11:14:42,992 DEBUG    model: entity
2019-05-16 11:14:42,994 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,017 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/index.json HTTP/1.1" 200 241
2019-05-16 11:14:43,052 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/status.json
2019-05-16 11:14:43,055 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/language.json
2019-05-16 11:14:43,061 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,064 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,076 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/rights.json
2019-05-16 11:14:43,077 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/topics.json
2019-05-16 11:14:43,080 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/format.json
2019-05-16 11:14:43,084 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/language.json HTTP/1.1" 200 1175
2019-05-16 11:14:43,084 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/status.json HTTP/1.1" 200 325
2019-05-16 11:14:43,085 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,088 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/genre.json
2019-05-16 11:14:43,094 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,097 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,097 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,099 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/public.json
2019-05-16 11:14:43,104 INFO     getting vocab: http://partner.densho.org/vocab/api/0.2/facility.json
2019-05-16 11:14:43,106 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,107 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 526398
2019-05-16 11:14:43,110 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/format.json HTTP/1.1" 200 715
2019-05-16 11:14:43,111 DEBUG    Starting new HTTP connection (1): partner.densho.org
2019-05-16 11:14:43,112 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/genre.json HTTP/1.1" 200 5348
2019-05-16 11:14:43,114 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/rights.json HTTP/1.1" 200 597
2019-05-16 11:14:43,118 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/public.json HTTP/1.1" 200 302
2019-05-16 11:14:43,127 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/facility.json HTTP/1.1" 200 48090
2019-05-16 11:14:43,261 INFO     Validating headers
2019-05-16 11:14:43,262 INFO     ok
2019-05-16 11:14:43,262 INFO     Validating rows
2019-05-16 11:14:43,340 INFO     ok
2019-05-16 11:14:43,340 INFO     ok
2019-05-16 11:14:43,341 INFO     Checking repository
2019-05-16 11:14:43,343 INFO     <git.Repo "/media/qnfs/kinkura/gold/ddr-testing-40227/.git">
2019-05-16 11:14:43,369 INFO     ok
2019-05-16 11:14:43,369 INFO     Confirming all entity IDs available
2019-05-16 11:14:43,370 DEBUG    idservice.IDServiceClient.check_eids(<DDR.identifier.Identifier collection:ddr-testing-40227>, 22 entity_ids)
2019-05-16 11:14:43,372 DEBUG    Starting new HTTPS connection (1): idservice.densho.org
2019-05-16 11:14:43,450 DEBUG    https://idservice.densho.org:443 "POST /api/0.1/objectids/ddr-testing-40227/check/ HTTP/1.1" 200 531
2019-05-16 11:14:43,453 INFO     200 OK
2019-05-16 11:14:43,453 INFO     0 registered
2019-05-16 11:14:43,453 INFO     22 NOT registered
2019-05-16 11:14:43,453 INFO     Checking for locally existing IDs
Traceback (most recent call last):
  File "/opt/ddr-local/venv/ddrlocal/bin/ddrimport", line 14, in <module>
    load_entry_point('ddr-cmdln==0.9.4b0', 'console_scripts', 'ddrimport')()
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 161, in entity
    csv_path, ci, config.VOCABS_URL, idservice_client
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 291, in run_checks
    chkeids = batch.Checker.check_eids(chkcsv['rowds'], ci, idservice_client)
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 265, in check_eids
    rowds, cidentifier.model, cidentifier.path_abs()
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 348, in _ids_in_local_repo
    already = [i for i in new_ids if i in existing_ids]
  File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/identifier.py", line 1070, in __eq__
    return self.path_abs() == other.path_abs()
AttributeError: 'unicode' object has no attribute 'path_abs'
pkikawa commented 5 years ago

oops, didnt mean to close

gjost commented 5 years ago

Should have clarified. That actual error is fixed in a commit I just pushed up (ddr-cmdln develop branch commit 3fbb975) but it's not in a package yet. Once that fix is in place you'll see all the data problems that led to the CSV fix.

pkikawa commented 5 years ago

mint

gjost commented 5 years ago

Fixed in ddr-cmdln commit 3fbb975 for package ddrlocal-develop_2.8.4a.

GeoffFroh commented 5 years ago

@pkikawa please test and close if fixed

pkikawa commented 5 years ago

tested and working on 284aDEV

pkikawa commented 5 years ago

regression on ddrlocal-master 285

(ddrlocal) ddr@DDREditor:/media/qnfs/kinkura/gold/ddr-testing-40231$ ddrimport entity -U philip.kikawa -P Qk325rrMx /media/sf_ddrshared/ddr-testing-40231-obj.csv /media/qnfs/kinkura/gold/ddr-testing-40231 2019-05-29 13:30:04,483 DEBUG 2019-05-29 13:30:04,483 DEBUG Username: philip.kikawa 2019-05-29 13:30:04,483 DEBUG Password: *** 2019-05-29 13:30:04,484 DEBUG idservice.IDServiceClient.login(philip.kikawa) 2019-05-29 13:30:04,486 DEBUG Starting new HTTPS connection (1): idservice.densho.org 2019-05-29 13:30:04,678 DEBUG https://idservice.densho.org:443 "POST /api/0.1/rest-auth/login/ HTTP/1.1" 200 50 2019-05-29 13:30:04,681 INFO Checking CSV file 2019-05-29 13:30:04,688 INFO 22 rows 2019-05-29 13:30:04,689 DEBUG Guessing model based on 22 rows 2019-05-29 13:30:04,689 DEBUG model: entity 2019-05-29 13:30:04,690 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,714 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/index.json HTTP/1.1" 200 241 2019-05-29 13:30:04,739 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/status.json 2019-05-29 13:30:04,742 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,742 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/language.json 2019-05-29 13:30:04,746 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,746 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/rights.json 2019-05-29 13:30:04,748 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/format.json 2019-05-29 13:30:04,748 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/facility.json 2019-05-29 13:30:04,749 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/topics.json 2019-05-29 13:30:04,750 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,750 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,752 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,754 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,754 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/public.json 2019-05-29 13:30:04,759 INFO getting vocab: http://partner.densho.org/vocab/api/0.2/genre.json 2019-05-29 13:30:04,760 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/status.json HTTP/1.1" 200 325 2019-05-29 13:30:04,757 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,763 DEBUG Starting new HTTP connection (1): partner.densho.org 2019-05-29 13:30:04,765 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/language.json HTTP/1.1" 200 1175 2019-05-29 13:30:04,765 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/rights.json HTTP/1.1" 200 597 2019-05-29 13:30:04,766 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/format.json HTTP/1.1" 200 715 2019-05-29 13:30:04,771 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 526398 2019-05-29 13:30:04,769 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/facility.json HTTP/1.1" 200 48090 2019-05-29 13:30:04,777 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/public.json HTTP/1.1" 200 302 2019-05-29 13:30:04,780 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/genre.json HTTP/1.1" 200 5348 2019-05-29 13:30:04,918 INFO Validating headers 2019-05-29 13:30:04,919 INFO ok 2019-05-29 13:30:04,919 INFO Validating rows 2019-05-29 13:30:04,996 INFO ok 2019-05-29 13:30:04,996 INFO ok 2019-05-29 13:30:04,996 INFO Checking repository 2019-05-29 13:30:05,000 INFO <git.Repo "/media/qnfs/kinkura/gold/ddr-testing-40231/.git"> 2019-05-29 13:30:05,027 INFO ok 2019-05-29 13:30:05,028 INFO Confirming all entity IDs available 2019-05-29 13:30:05,028 DEBUG idservice.IDServiceClient.check_eids(, 22 entity_ids) 2019-05-29 13:30:05,030 DEBUG Starting new HTTPS connection (1): idservice.densho.org 2019-05-29 13:30:05,111 DEBUG https://idservice.densho.org:443 "POST /api/0.1/objectids/ddr-testing-40231/check/ HTTP/1.1" 200 531 2019-05-29 13:30:05,113 INFO 200 OK 2019-05-29 13:30:05,113 INFO 0 registered 2019-05-29 13:30:05,113 INFO 22 NOT registered 2019-05-29 13:30:05,113 INFO Checking for locally existing IDs Traceback (most recent call last): File "/opt/ddr-local/venv/ddrlocal/bin/ddrimport", line 14, in load_entry_point('ddr-cmdln==0.9.4b0', 'console_scripts', 'ddrimport')() File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 722, in call return self.main(*args, kwargs) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, *ctx.params) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 161, in entity csv_path, ci, config.VOCABS_URL, idservice_client File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/cli/ddrimport.py", line 291, in run_checks chkeids = batch.Checker.check_eids(chkcsv['rowds'], ci, idservice_client) File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 265, in check_eids rowds, cidentifier.model, cidentifier.path_abs() File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/batch.py", line 348, in _ids_in_local_repo already = [i for i in new_ids if i in existing_ids] File "/opt/ddr-local/venv/ddrlocal/lib/python2.7/site-packages/ddr_cmdln-0.9.4b0-py2.7.egg/DDR/identifier.py", line 1070, in eq return self.path_abs() == other.path_abs() AttributeError: 'unicode' object has no attribute 'path_abs'

gjost commented 5 years ago

Should be fixed in package ddrlocal-master_2.8.6.

pkikawa commented 5 years ago

working in 286MA