search

denshoproject / ddr-cmdln

Command-line tools for automating the Densho Digital Repository's various processes.
Other
0 stars 2 forks source link

ddrimport cannot import segments #86

Closed gjost closed 6 years ago

gjost commented 6 years ago

ddrimport cannot import the following CSV

"id","record_created","record_lastmod","status","public","sort","title","description","creation","location","creators","language","genre","format","extent","contributor","alternate_id","digitize_person","digitize_organization","digitize_date","credit","rights","rights_statement","topics","persons","facility","chronology","geography","parent","signature_id","notes"
"ddr-testing-40088-1","2016-11-02T15:58:17PDT-0700","2018-06-15T16:04:47PDT-0700","completed","1","1","Gene Akutsu Interview I","Nisei male. Born September 23, 1925, in Seattle, Washington. Spent prewar childhood in Seattle's Nihonmachi. Incarcerated at Puyallup Assembly Center, Washington, and Minidoka concentration camp, Idaho. Refused to participate in draft, imprisoned at McNeil Island Penitentiary, Washington, for resisting the draft. Resettled in Seattle. Mr. Akutsu passed away on October 6, 2011.","July 25, 1997","Seattle, Washington","Gene Akutsu [1]:narrator; Larry Hashima:interviewer; Stephen Fugita:interviewer; Matt Emery:videographer","eng","interview","vh","02:47:34","Densho","[denshouid: denshovh-agene-01]","Dana Hoshide","Densho","2004-03-25 00:00:00.0","Courtesy of Densho","cc","","term:Identity and values: Nisei|id:44;term:Geographic communities: Washington: Seattle|id:293;term:World War II: Resistance and dissidence: Draft resistance|id:95","","term:Puyallup (Camp Harmony)|id:11;term:Minidoka|id:8;term:McNeil Island Penitentiary|id:33","","","","denshovh-agene-01","[int_notes:];[technotes:B-roll footage at the end of tape 3];[capture_format: CanopusDV-AVI];[master_media_type:DVC];[process_notes:]"
"ddr-testing-40088-1-1","2016-11-02T16:02:46PDT-0700","2018-06-14T10:11:16PDT-0700","completed","1","1","Gene Akutsu Segment 1","Immigrant parent's arrival in U.S. and means of making a living","July 25, 1997","Seattle, Washington","Gene Akutsu [1]:narrator; Larry Hashima:interviewer; Stephen Fugita:interviewer; Matt Emery:videographer","eng","interview","vh","00:03:44","Densho","[denshouid: denshovh-agene-01-0001]","Dana Hoshide","Densho","2004-03-25 00:00:00.0","Courtesy of Densho","cc","","term:Immigration and citizenship: Arrival|id:4","","","startdate:1900-01-01 00:00:00.0|term:1900-1909","term:Seattle, Washington|id:""http://vocab.getty.edu/tgn/7014494""","ddr-testing-40088-1","denshovh-agene-01","[int_notes:];[technotes:B-roll footage at the end of tape 3];[capture_format: CanopusDV-AVI];[physmedia_time_in:1:17:29];[physmedia_time_out:1:21:14];[physmedia_id:1713]"
"ddr-testing-40088-1-2","2016-11-02T16:02:46PDT-0700","2018-06-14T10:12:13PDT-0700","completed","1","2","Gene Akutsu Segment 2","Growing up around ""Profanity Hill,"" Seattle, Washington","July 25, 1997","Seattle, Washington","Gene Akutsu [1]:narrator; Larry Hashima:interviewer; Stephen Fugita:interviewer; Matt Emery:videographer","eng","interview","vh","00:03:01","Densho","[denshouid: denshovh-agene-01-0002]","Dana Hoshide","Densho","2004-03-25 00:00:00.0","Courtesy of Densho","cc","","term:Geographic communities: Washington: Seattle|id:293","","","startdate:1880-01-01 00:00:00.0|term:Prewar","term:Seattle, Washington|id:""http://vocab.getty.edu/tgn/7014494""","ddr-testing-40088-1","denshovh-agene-01","[int_notes:];[technotes:B-roll footage at the end of tape 3];[capture_format: CanopusDV-AVI];[physmedia_time_in:1:21:14];[physmedia_time_out:1:24:15];[physmedia_id:1713]"
"ddr-testing-40088-1-3","2016-11-02T16:02:46-0700","2016-11-02T15:58:33PDT-0700","completed","1","3","Gene Akutsu Segment 3","Pearl Harbor: A teenager's memories of what he understood and how he felt","July 25, 1997","Seattle, Washington","Gene Akutsu [1]:narrator; Larry Hashima:interviewer; Stephen Fugita:interviewer; Matt Emery:videographer","eng","interview","vh","00:02:19","Densho","[denshouid: denshovh-agene-01-0003]","Dana Hoshide","Densho","2004-03-25 00:00:00.0","Courtesy of Densho","cc","","term:World War II: Pearl Harbor and aftermath: Personal recollections|id:51","","","startdate:1941-01-01 00:00:00.0|term:1941","","ddr-testing-40088-1","denshovh-agene-01","[int_notes:];[technotes:B-roll footage at the end of tape 3];[capture_format: CanopusDV-AVI];[physmedia_time_in:1:24:15];[physmedia_time_out:1:26:36];[physmedia_id:1713]"

It imports the first entity fine but can't import the segments.

(ddrlocal)ddr@denshodeb8:/opt/ddr-local$ ddrimport entity -u gjost -m gjost@densho.org -U gjost /tmp/ddr-testing-40088-entity.csv /var/www/media/ddr/dd
r-testing-40088/
2018-08-10 12:53:00,357 DEBUG    <DDR.identifier.Identifier collection:ddr-testing-40088>
2018-08-10 12:53:00,359 DEBUG    Username: gjost
Password: 
2018-08-10 12:53:02,917 DEBUG    idservice.IDServiceClient.login(gjost)
2018-08-10 12:53:02,927 DEBUG    Starting new HTTPS connection (1): idservice.densho.org:443
2018-08-10 12:53:03,319 DEBUG    https://idservice.densho.org:443 "POST /api/0.1/rest-auth/login/ HTTP/1.1" 200 50
2018-08-10 12:53:03,328 INFO     Checking CSV file
2018-08-10 12:53:03,332 INFO     4 rows
2018-08-10 12:53:03,332 DEBUG    Guessing model based on 4 rows
2018-08-10 12:53:03,332 DEBUG    model: entity
2018-08-10 12:53:03,333 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/language.json
2018-08-10 12:53:03,334 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/format.json
2018-08-10 12:53:03,334 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/rights.json
2018-08-10 12:53:03,335 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/topics.json
2018-08-10 12:53:03,347 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/facility.json
2018-08-10 12:53:03,349 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/status.json
2018-08-10 12:53:03,350 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/public.json
2018-08-10 12:53:03,350 INFO     getting vocab: /opt/ddr-local/ddr-vocab/api/0.2/genre.json
2018-08-10 12:53:03,351 INFO     Validating headers
2018-08-10 12:53:03,351 INFO     ok
2018-08-10 12:53:03,352 INFO     Validating rows
2018-08-10 12:53:03,385 INFO     ok
2018-08-10 12:53:03,385 ERROR    FAIL
2018-08-10 12:53:03,386 INFO     Checking repository
2018-08-10 12:53:03,388 INFO     <git.Repo "/var/www/media/ddr/ddr-testing-40088/.git">
2018-08-10 12:53:03,514 INFO     ok
2018-08-10 12:53:03,514 INFO     Confirming all entity IDs available
2018-08-10 12:53:03,514 DEBUG    idservice.IDServiceClient.check_eids(<DDR.identifier.Identifier collection:ddr-testing-40088>, 4 entity_ids)
2018-08-10 12:53:03,517 DEBUG    Starting new HTTPS connection (1): idservice.densho.org:443
2018-08-10 12:53:03,801 DEBUG    https://idservice.densho.org:443 "POST /api/0.1/objectids/ddr-testing-40088/check/ HTTP/1.1" 200 127
2018-08-10 12:53:03,809 INFO     200 OK
2018-08-10 12:53:03,810 INFO     1 registered
2018-08-10 12:53:03,812 INFO     3 NOT registered
2018-08-10 12:53:03,814 INFO     Checking for locally existing IDs
2018-08-10 12:53:03,818 DEBUG    0 locally existing
2018-08-10 12:53:03,820 ERROR    FAIL
2018-08-10 12:53:03,820 ERROR    TESTS FAILED--QUITTING!

It can be forced to import the segments by using the --nocheck flag.

gjost commented 6 years ago

See densho/ddr-idservice#4

gjost commented 6 years ago

Fixed in fdfd6ae