search

denshoproject / ddr-cmdln

Command-line tools for automating the Densho Digital Repository's various processes.
Other
0 stars 2 forks source link

ddr-import does not update 'record_lastmod' attribute #76

Closed GeoffFroh closed 6 years ago

GeoffFroh commented 6 years ago

ddr-import does not update the 'record_lastmod' attribute when it modifies a collection or entity json file. It should definitely do this.

gjost commented 6 years ago

I exported and then reimported some entity data in a test collection and it worked for me. Can you provide more details?

gjost commented 6 years ago

See https://github.com/densho/ddr-cmdln/issues/78

gjost commented 6 years ago

What branch and version are you running? I just tried this again and it behaved like I'd expect.

$ ddr-export entity /var/www/media/ddr/ddr-testing-40080 /tmp/ddr-testing-40080-entity.csv
... modified CSV file ...
$ ddr-import entity /tmp/ddr-testing-40080-entity-mod.csv /var/www/media/ddr/ddr-testing-40080
$ cd /var/www/media/ddr/ddr-testing-40080
$ git diffc files/ddr-testing-40080-1/entity.json
diff --git a/files/ddr-testing-40080-1/entity.json b/files/ddr-testing-40080-1/entity.json
index 78c5723..adedd44 100644
--- a/files/ddr-testing-40080-1/entity.json
+++ b/files/ddr-testing-40080-1/entity.json
@@ -1,6 +1,6 @@
 [
     {
-        "app_commit": "f9f6c7a214e682342216d047b101d2f26f7fd3f0  (HEAD, origin/master, origin/HEAD, master) 2018-06-21 11:23:38 -0700",
+        "app_commit": "bac19d6d051dc3b62cebac48a2a1c901f2d00c14  (HEAD, origin/master, origin/HEAD, master) 2018-06-28 13:46:55 -0700",
         "app_release": "0.9.4-beta",
         "application": "https://github.com/densho/ddr-cmdln.git",
         "defs_path": "/opt/ddr-local/ddr-defs/repo_models/entity.py",
@@ -14,22 +14,22 @@
         "record_created": "2018-06-22T12:44:20PDT-0700"
     },
     {
-        "record_lastmod": "2018-06-22T12:45:15PDT-0700"
+        "record_lastmod": "2018-06-28T17:28:06PDT-0700"
     },
     {
-        "status": "inprocess"
+        "status": "completed"
     },
     {
         "public": "1"
     },
     {
-        "sort": 1
+        "sort": "1"
     },
     {
         "title": "Supermicro"
     },
     {
-        "description": ""
+        "description": "nothing much to say about this actually"
...

Could I get a bit more detail on exactly what you're doing, what you're expecting to see and what you're actually seeing that's different from what you expect, and maybe some of the data you're using so I can duplicate this?

GeoffFroh commented 6 years ago

ddr-local: 3f3b13fcfd04f6228979f172e3dff58f4efd96a4 ddr-cmdln: f9f6c7a214e682342216d047b101d2f26f7fd3f0 ddr-defs: 4d83d8c372f9fbb011be4b21412d625533eea28a

I found the issue with another repo, but I tested twice on a clone from mits of ddr-densho-1001 with this import csv: ddr-densho-1001-41-ddrfiles-notrans.csv.txt

The ddr-import command and output:

(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$ ddr-import file /media/qnfs/kinkura/operations/vhprocess/out/ddr-files/ddr-densho-1001-41-ddrfiles-notrans.csv /media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001
2018-06-29 09:24:13,474 DEBUG    <DDR.identifier.Identifier collection:ddr-densho-1001>
2018-06-29 09:24:13,475 DEBUG    /media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001
2018-06-29 09:24:13,475 INFO     batch import files ----------------------------
2018-06-29 09:24:13,477 DEBUG    csv_dir /media/qnfs/kinkura/operations/vhprocess/out/ddr-files
2018-06-29 09:24:13,477 DEBUG    entity_class <class 'DDR.models.entity.Entity'>
2018-06-29 09:24:13,478 DEBUG    <git.Repo "/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001/.git">
2018-06-29 09:24:13,478 INFO     Reading /media/qnfs/kinkura/operations/vhprocess/out/ddr-files/ddr-densho-1001-41-ddrfiles-notrans.csv
2018-06-29 09:24:13,479 INFO     11 rows
2018-06-29 09:24:13,480 INFO     csv_load rowds
2018-06-29 09:24:13,494 DEBUG    getting topics: http://partner.densho.org/vocab/api/0.2/topics.json
2018-06-29 09:24:13,497 DEBUG    Starting new HTTP connection (1): partner.densho.org
2018-06-29 09:24:13,737 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 439377
2018-06-29 09:24:14,025 DEBUG    ok
2018-06-29 09:24:14,118 DEBUG    getting topics: http://partner.densho.org/vocab/api/0.2/topics.json
2018-06-29 09:24:14,120 DEBUG    Starting new HTTP connection (1): partner.densho.org
2018-06-29 09:24:14,135 DEBUG    http://partner.densho.org:80 "GET /vocab/api/0.2/topics.json HTTP/1.1" 200 439377
2018-06-29 09:24:14,386 DEBUG    ok
2018-06-29 09:24:14,464 INFO     - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:14,464 INFO     Updating existing files
2018-06-29 09:24:14,465 DEBUG    0 updated in 0:00:00.000075
2018-06-29 09:24:14,465 INFO     - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:14,465 INFO     Adding new files
2018-06-29 09:24:14,465 INFO     + 1/11 - ddr-densho-1001-41-3 (ddr-densho-1001-41-3-mezzanine-9ae9512ee7.mpg)
2018-06-29 09:24:14,465 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-3>
2018-06-29 09:24:14,753 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-3-mezzanine-9ae9512ee7>
2018-06-29 09:24:14,753 DEBUG    | 0:00:00.286975
2018-06-29 09:24:14,753 INFO     + 2/11 - ddr-densho-1001-41-6 (ddr-densho-1001-41-6-mezzanine-396d543bfd.mpg)
2018-06-29 09:24:14,754 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-6>
2018-06-29 09:24:15,029 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-6-mezzanine-396d543bfd>
2018-06-29 09:24:15,029 DEBUG    | 0:00:00.274930
2018-06-29 09:24:15,030 INFO     + 3/11 - ddr-densho-1001-41-7 (ddr-densho-1001-41-7-mezzanine-90e9ce6158.mpg)
2018-06-29 09:24:15,030 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-7>
2018-06-29 09:24:15,299 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-7-mezzanine-90e9ce6158>
2018-06-29 09:24:15,300 DEBUG    | 0:00:00.268878
2018-06-29 09:24:15,300 INFO     + 4/11 - ddr-densho-1001-41-2 (ddr-densho-1001-41-2-mezzanine-6e4b9fbf77.mpg)
2018-06-29 09:24:15,300 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-2>
2018-06-29 09:24:15,533 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-2-mezzanine-6e4b9fbf77>
2018-06-29 09:24:15,534 DEBUG    | 0:00:00.232850
2018-06-29 09:24:15,534 INFO     + 5/11 - ddr-densho-1001-41-10 (ddr-densho-1001-41-10-mezzanine-8a5d16616c.mpg)
2018-06-29 09:24:15,535 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-10>
2018-06-29 09:24:15,785 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-10-mezzanine-8a5d16616c>
2018-06-29 09:24:15,786 DEBUG    | 0:00:00.250527
2018-06-29 09:24:15,786 INFO     + 6/11 - ddr-densho-1001-41-4 (ddr-densho-1001-41-4-mezzanine-543bc163c7.mpg)
2018-06-29 09:24:15,786 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-4>
2018-06-29 09:24:16,023 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-4-mezzanine-543bc163c7>
2018-06-29 09:24:16,023 DEBUG    | 0:00:00.236539
2018-06-29 09:24:16,024 INFO     + 7/11 - ddr-densho-1001-41-9 (ddr-densho-1001-41-9-mezzanine-bbf0d0adb3.mpg)
2018-06-29 09:24:16,024 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-9>
2018-06-29 09:24:16,315 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-9-mezzanine-bbf0d0adb3>
2018-06-29 09:24:16,316 DEBUG    | 0:00:00.291136
2018-06-29 09:24:16,316 INFO     + 8/11 - ddr-densho-1001-41-8 (ddr-densho-1001-41-8-mezzanine-65a4806384.mpg)
2018-06-29 09:24:16,316 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-8>
2018-06-29 09:24:16,579 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-8-mezzanine-65a4806384>
2018-06-29 09:24:16,580 DEBUG    | 0:00:00.262223
2018-06-29 09:24:16,580 INFO     + 9/11 - ddr-densho-1001-41-1 (ddr-densho-1001-41-1-mezzanine-d1771706aa.mpg)
2018-06-29 09:24:16,580 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-1>
2018-06-29 09:24:16,839 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-1-mezzanine-d1771706aa>
2018-06-29 09:24:16,840 DEBUG    | 0:00:00.258705
2018-06-29 09:24:16,840 INFO     + 10/11 - ddr-densho-1001-41-5 (ddr-densho-1001-41-5-mezzanine-b443d48cc9.mpg)
2018-06-29 09:24:16,840 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-5>
2018-06-29 09:24:17,095 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-5-mezzanine-b443d48cc9>
2018-06-29 09:24:17,096 DEBUG    | 0:00:00.254548
2018-06-29 09:24:17,096 INFO     + 11/11 - ddr-densho-1001-41-11 (ddr-densho-1001-41-11-mezzanine-af5d0612ab.mpg)
2018-06-29 09:24:17,096 DEBUG    | parent <DDR.models.entity.Entity segment:ddr-densho-1001-41-11>
2018-06-29 09:24:17,366 DEBUG    |   file <DDR.models.files.File file:ddr-densho-1001-41-11-mezzanine-af5d0612ab>
2018-06-29 09:24:17,366 DEBUG    | 0:00:00.269253
2018-06-29 09:24:17,366 DEBUG    11 added in 0:00:02.901191
2018-06-29 09:24:17,367 INFO     - - - - - - - - - - - - - - - - - - - - - - - -
2018-06-29 09:24:17,369 INFO     DONE - 0:00:03.895870 elapsed

And here's the diff of one of the edited entities:

(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$ git diff --cached files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
diff --git a/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json b/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
index 12d812e6..1eaa77ca 100644
--- a/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
+++ b/files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/entity.json
@@ -1,11 +1,11 @@
 [
     {
-        "app_commit": "452cdfd6692914207c9bdee38ca17aceece2beac  (HEAD -> develop, origin/develop) 2018-05-24 11:18:51 -0700",
+        "app_commit": "f9f6c7a214e682342216d047b101d2f26f7fd3f0  (HEAD -> master, origin/master, origin/HEAD) 2018-06-21 11:23:38 -0700",
         "app_release": "0.9.4-beta",
         "application": "https://github.com/densho/ddr-cmdln.git",
         "defs_path": "/opt/ddr-local/ddr-defs/repo_models/segment.py",
-        "git_version": "git version 2.11.0; git-annex version: 6.20170101.1\nbuild flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi\nkey/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL\nremote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external\nlocal repository version: 5\nsupported repository versions: 3 5 6\nupgrade supported from repository versions: 0 1 2 3 4 5\noperating system: linux x86_64",
-        "models_commit": "8c99c801a5a1a369ebbdd6138d346985cfbdc074  (HEAD -> develop, origin/develop) 2018-05-29 16:21:37 -0700"
+        "git_version": "git version 2.11.0; git-annex version: 6.20170101.1\nbuild flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi\nkey/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL\nremote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external\nlocal repository version: unknown\nsupported repository versions: 3 5 6\nupgrade supported from repository versions: 0 1 2 3 4 5\noperating system: linux x86_64",
+        "models_commit": "4d83d8c372f9fbb011be4b21412d625533eea28a  (HEAD -> master, origin/master, origin/HEAD) 2018-06-20 20:10:06 -0700"
     },
     {
         "id": "ddr-densho-1001-41-1"
@@ -130,9 +130,22 @@
             {
                 "files": [
                     {
+                        "id": "ddr-densho-1001-41-1-mezzanine-d1771706aa",
+                        "label": "Grant Hirabayashi Interview Segment 1",
+                        "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-mezzanine-d1771706aa.mpg",
+                        "public": "1",
+                        "size": "485158916",
+                        "sort": "1"
+                    }
+                ],
+                "role": "mezzanine"
+            },
+            {
+                "files": [
+                    {
                         "id": "ddr-densho-1001-41-1-transcript-ef493b74e2",
                         "label": "Grant Hirabayashi Interview Segment 1 Transcript",
-                        "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-transcript-ef493b74e2",
+                        "path_rel": "files/ddr-densho-1001-41/files/ddr-densho-1001-41-1/files/ddr-densho-1001-41-1-transcript-ef493b74e2.htm",
                         "public": "1",
                         "size": 5020,
                         "sort": 1
(ddrlocal) ddr@maunakea:/media/qnfs/kinkura/temp/20180629-ddr-densho-1001/ddr-densho-1001$

Note that the record_lastmod attribute was not modified.

gjost commented 6 years ago

Ahh, so you're doing file operations but record_lastmod on the parent entity is not being touched. That's a different use-case from what I was testing.