Closed GeoffFroh closed 6 years ago
I exported and then reimported some entity data in a test collection and it worked for me. Can you provide more details?
What branch and version are you running? I just tried this again and it behaved like I'd expect.
$ ddr-export entity /var/www/media/ddr/ddr-testing-40080 /tmp/ddr-testing-40080-entity.csv
... modified CSV file ...
$ ddr-import entity /tmp/ddr-testing-40080-entity-mod.csv /var/www/media/ddr/ddr-testing-40080
$ cd /var/www/media/ddr/ddr-testing-40080
$ git diffc files/ddr-testing-40080-1/entity.json
diff --git a/files/ddr-testing-40080-1/entity.json b/files/ddr-testing-40080-1/entity.json
index 78c5723..adedd44 100644
--- a/files/ddr-testing-40080-1/entity.json
+++ b/files/ddr-testing-40080-1/entity.json
@@ -1,6 +1,6 @@
[
{
- "app_commit": "f9f6c7a214e682342216d047b101d2f26f7fd3f0 (HEAD, origin/master, origin/HEAD, master) 2018-06-21 11:23:38 -0700",
+ "app_commit": "bac19d6d051dc3b62cebac48a2a1c901f2d00c14 (HEAD, origin/master, origin/HEAD, master) 2018-06-28 13:46:55 -0700",
"app_release": "0.9.4-beta",
"application": "https://github.com/densho/ddr-cmdln.git",
"defs_path": "/opt/ddr-local/ddr-defs/repo_models/entity.py",
@@ -14,22 +14,22 @@
"record_created": "2018-06-22T12:44:20PDT-0700"
},
{
- "record_lastmod": "2018-06-22T12:45:15PDT-0700"
+ "record_lastmod": "2018-06-28T17:28:06PDT-0700"
},
{
- "status": "inprocess"
+ "status": "completed"
},
{
"public": "1"
},
{
- "sort": 1
+ "sort": "1"
},
{
"title": "Supermicro"
},
{
- "description": ""
+ "description": "nothing much to say about this actually"
...
Could I get a bit more detail on exactly what you're doing, what you're expecting to see and what you're actually seeing that's different from what you expect, and maybe some of the data you're using so I can duplicate this?
ddr-local: 3f3b13fcfd04f6228979f172e3dff58f4efd96a4 ddr-cmdln: f9f6c7a214e682342216d047b101d2f26f7fd3f0 ddr-defs: 4d83d8c372f9fbb011be4b21412d625533eea28a
I found the issue with another repo, but I tested twice on a clone from mits of ddr-densho-1001 with this import csv: ddr-densho-1001-41-ddrfiles-notrans.csv.txt
The ddr-import
command and output:
(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$ ddr-import file /media/qnfs/kinkura/operations/vhprocess/out/ddr-files/ddr-densho-1001-41-ddrfiles-notrans.csv /media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001
2018-06-29 09:24:13,474 DEBUG <DDR.identifier.Identifier collection:ddr-densho-1001>
2018-06-29 09:24:13,475 DEBUG /media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001
2018-06-29 09:24:13,475 INFO batch import files ----------------------------
2018-06-29 09:24:13,477 DEBUG csv_dir /media/qnfs/kinkura/operations/vhprocess/out/ddr-files
2018-06-29 09:24:13,477 DEBUG entity_class <class 'DDR.models.entity.Entity'>
2018-06-29 09:24:13,478 DEBUG <git.Repo "/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001/.git">
2018-06-29 09:24:13,478 INFO Reading /media/qnfs/kinkura/operations/vhprocess/out/ddr-files/ddr-densho-1001-41-ddrfiles-notrans.csv
2018-06-29 09:24:13,479 INFO 11 rows
2018-06-29 09:24:13,480 INFO csv_load rowds
2018-06-29 09:24:13,494 DEBUG getting topics: http://partner.densho.org/vocab/api/0.2/topics.json
2018-06-29 09:24:13,497 DEBUG Starting new HTTP connection (1): partner.densho.org
2018-06-29 09:24:13,737 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 439377
2018-06-29 09:24:14,025 DEBUG ok
2018-06-29 09:24:14,118 DEBUG getting topics: http://partner.densho.org/vocab/api/0.2/topics.json
2018-06-29 09:24:14,120 DEBUG Starting new HTTP connection (1): partner.densho.org
2018-06-29 09:24:14,135 DEBUG http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 439377
2018-06-29 09:24:14,386 DEBUG ok
2018-06-29 09:24:14,464 INFO - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:14,464 INFO Updating existing files
2018-06-29 09:24:14,465 DEBUG 0 updated in 0:00:00.000075
2018-06-29 09:24:14,465 INFO - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:14,465 INFO Adding new files
2018-06-29 09:24:14,465 INFO + 1/11 - ddr-densho-1001-41-3 (ddr-densho-1001-41-3-mezzanine-9ae9512ee7.mpg)
2018-06-29 09:24:14,465 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-3>
2018-06-29 09:24:14,753 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-3-mezzanine-9ae9512ee7>
2018-06-29 09:24:14,753 DEBUG | 0:00:00.286975
2018-06-29 09:24:14,753 INFO + 2/11 - ddr-densho-1001-41-6 (ddr-densho-1001-41-6-mezzanine-396d543bfd.mpg)
2018-06-29 09:24:14,754 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-6>
2018-06-29 09:24:15,029 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-6-mezzanine-396d543bfd>
2018-06-29 09:24:15,029 DEBUG | 0:00:00.274930
2018-06-29 09:24:15,030 INFO + 3/11 - ddr-densho-1001-41-7 (ddr-densho-1001-41-7-mezzanine-90e9ce6158.mpg)
2018-06-29 09:24:15,030 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-7>
2018-06-29 09:24:15,299 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-7-mezzanine-90e9ce6158>
2018-06-29 09:24:15,300 DEBUG | 0:00:00.268878
2018-06-29 09:24:15,300 INFO + 4/11 - ddr-densho-1001-41-2 (ddr-densho-1001-41-2-mezzanine-6e4b9fbf77.mpg)
2018-06-29 09:24:15,300 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-2>
2018-06-29 09:24:15,533 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-2-mezzanine-6e4b9fbf77>
2018-06-29 09:24:15,534 DEBUG | 0:00:00.232850
2018-06-29 09:24:15,534 INFO + 5/11 - ddr-densho-1001-41-10 (ddr-densho-1001-41-10-mezzanine-8a5d16616c.mpg)
2018-06-29 09:24:15,535 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-10>
2018-06-29 09:24:15,785 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-10-mezzanine-8a5d16616c>
2018-06-29 09:24:15,786 DEBUG | 0:00:00.250527
2018-06-29 09:24:15,786 INFO + 6/11 - ddr-densho-1001-41-4 (ddr-densho-1001-41-4-mezzanine-543bc163c7.mpg)
2018-06-29 09:24:15,786 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-4>
2018-06-29 09:24:16,023 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-4-mezzanine-543bc163c7>
2018-06-29 09:24:16,023 DEBUG | 0:00:00.236539
2018-06-29 09:24:16,024 INFO + 7/11 - ddr-densho-1001-41-9 (ddr-densho-1001-41-9-mezzanine-bbf0d0adb3.mpg)
2018-06-29 09:24:16,024 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-9>
2018-06-29 09:24:16,315 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-9-mezzanine-bbf0d0adb3>
2018-06-29 09:24:16,316 DEBUG | 0:00:00.291136
2018-06-29 09:24:16,316 INFO + 8/11 - ddr-densho-1001-41-8 (ddr-densho-1001-41-8-mezzanine-65a4806384.mpg)
2018-06-29 09:24:16,316 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-8>
2018-06-29 09:24:16,579 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-8-mezzanine-65a4806384>
2018-06-29 09:24:16,580 DEBUG | 0:00:00.262223
2018-06-29 09:24:16,580 INFO + 9/11 - ddr-densho-1001-41-1 (ddr-densho-1001-41-1-mezzanine-d1771706aa.mpg)
2018-06-29 09:24:16,580 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-1>
2018-06-29 09:24:16,839 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-1-mezzanine-d1771706aa>
2018-06-29 09:24:16,840 DEBUG | 0:00:00.258705
2018-06-29 09:24:16,840 INFO + 10/11 - ddr-densho-1001-41-5 (ddr-densho-1001-41-5-mezzanine-b443d48cc9.mpg)
2018-06-29 09:24:16,840 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-5>
2018-06-29 09:24:17,095 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-5-mezzanine-b443d48cc9>
2018-06-29 09:24:17,096 DEBUG | 0:00:00.254548
2018-06-29 09:24:17,096 INFO + 11/11 - ddr-densho-1001-41-11 (ddr-densho-1001-41-11-mezzanine-af5d0612ab.mpg)
2018-06-29 09:24:17,096 DEBUG | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-11>
2018-06-29 09:24:17,366 DEBUG | file <DDR.models.files.File file:ddr-densho-1001-41-11-mezzanine-af5d0612ab>
2018-06-29 09:24:17,366 DEBUG | 0:00:00.269253
2018-06-29 09:24:17,366 DEBUG 11 added in 0:00:02.901191
2018-06-29 09:24:17,367 INFO - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:17,369 INFO DONE - 0:00:03.895870 elapsed
And here's the diff of one of the edited entities:
(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$ git diff --cached files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
diff --git a/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json b/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
index 12d812e6..1eaa77ca 100644
--- a/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
+++ b/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
@@ -1,11 +1,11 @@
[
{
- "app_commit": "452cdfd6692914207c9bdee38ca17aceece2beac (HEAD -> develop, origin/develop) 2018-05-24 11:18:51 -0700",
+ "app_commit": "f9f6c7a214e682342216d047b101d2f26f7fd3f0 (HEAD -> master, origin/master, origin/HEAD) 2018-06-21 11:23:38 -0700",
"app_release": "0.9.4-beta",
"application": "https://github.com/densho/ddr-cmdln.git",
"defs_path": "/opt/ddr-local/ddr-defs/repo_models/segment.py",
- "git_version": "git version 2.11.0; git-annex version: 6.20170101.1\nbuild flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi\nkey/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL\nremote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external\nlocal repository version: 5\nsupported repository versions: 3 5 6\nupgrade supported from repository versions: 0 1 2 3 4 5\noperating system: linux x86_64",
- "models_commit": "8c99c801a5a1a369ebbdd6138d346985cfbdc074 (HEAD -> develop, origin/develop) 2018-05-29 16:21:37 -0700"
+ "git_version": "git version 2.11.0; git-annex version: 6.20170101.1\nbuild flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi\nkey/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL\nremote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external\nlocal repository version: unknown\nsupported repository versions: 3 5 6\nupgrade supported from repository versions: 0 1 2 3 4 5\noperating system: linux x86_64",
+ "models_commit": "4d83d8c372f9fbb011be4b21412d625533eea28a (HEAD -> master, origin/master, origin/HEAD) 2018-06-20 20:10:06 -0700"
},
{
"id": "ddr-densho-1001-41-1"
@@ -130,9 +130,22 @@
{
"files": [
{
+ "id": "ddr-densho-1001-41-1-mezzanine-d1771706aa",
+ "label": "Grant Hirabayashi Interview Segment 1",
+ "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-mezzanine-d1771706aa.mpg",
+ "public": "1",
+ "size": "485158916",
+ "sort": "1"
+ }
+ ],
+ "role": "mezzanine"
+ },
+ {
+ "files": [
+ {
"id": "ddr-densho-1001-41-1-transcript-ef493b74e2",
"label": "Grant Hirabayashi Interview Segment 1 Transcript",
- "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-transcript-ef493b74e2",
+ "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-transcript-ef493b74e2.htm",
"public": "1",
"size": 5020,
"sort": 1
(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$
Note that the record_lastmod
attribute was not modified.
Ahh, so you're doing file operations but record_lastmod
on the parent entity is not being touched. That's a different use-case from what I was testing.
ddr-import does not update the 'record_lastmod' attribute when it modifies a collection or entity json file. It should definitely do this.