Release 2.72

[gamontoya]

Descriptive summary

Version: 2.72 Release Manager: @lsitu Product Owner and Reviewers: @gamontoya Operations: @jhriv or @rstanonik

Dependency

private_config Release 2.1: https://github.com/ucsdlib/private_config/issues/18

---

To Verify

• [ ] CIL metadata transformation QA
• [ ] CIL Harvesting - Retrieve initial batch of master source files from CIL's Public REST API
• [ ] VRR PDF watermarking: Use python script
• [ ] VRR image watermarking: Use ImageMagick
• [ ] Add to DAMS Manager to trigger watermarking for VRR pdfs/images
• [ ] Batch metadata overlay: deliverable 1
• [ ] Batch metadata overlay: deliverable 2
• [ ] Batch metadata overlay: deliverable 3
• [ ] Batch metadata overlay: deliverable 4
• [x] Give dams-curator group access to the DAMS Collection Status Report and File Report Tool
• [x] Improve output of the DAMS Collection Status Report

[gamontoya]

@lsitu Please reference the tickets that need to be verified above.

[lsitu]

@gamontoya / @hjsyoo / @abbpenn93 / @mdpeters DAMS Manager release 2.72 is deployed to staging. The support for PDFs and images watermarking is ready for review now. I've started the CIL Harvesting process to generate the results for QA, which will download the JSON sources (as well as the missing and new videos) from the CIL public API, which could be done in a day or two. I'll let you know once it's done and ready. Thanks.

[mdpeters]

@lsitu pdf watermarking appears to be working I'm going to do some more testing with various sized pdfs to confirm. Image watermarking is sort of working, the image is getting watermarked but both the image-preview and image-service images are getting a use value of document-service and are not displaying in the DAMS object view.

[lsitu]

@mdpeters Could you give me the links to the objects that you are testing so that I can take a look? Thank you.

[mdpeters]

@lsitu sure, meant to do that before: Image https://librarytest.ucsd.edu/dc/object/bd78717010 (ignore the title, it was set to curator only) PDF https://librarytest.ucsd.edu/dc/object/bd47317414 this worked as expected though it does look like the watermark isn't sizing down correctly when aligned vertically, it does seem to size down correctly when aligned horizontally, pages 14 and 15 illustrate the sizing working correctly, on all of the vertical pages it's getting cut off at the bottom

[lsitu]

@mppeters Thanks. I'll fix that file use value with document-service issue.

For the PDF watermarking, we are using the Python script https://github.com/ucsdlib/watermark. @rstanonik / @dlasusa Do you have any thoughts on fixing the watermarking image vertical scaling issue in PDF that @mdpeters reported above (https://github.com/ucsdlib/damsmanager/issues/339#issuecomment-507821930)? Thank you.

[lsitu]

@gamontoya / @hjsyoo / @abbypenn93 The CIL Harvesting process is done and the transformed CSV output is ready for QA now. Note that the support for downloading JSON sources (as well as the missing and new videos) from the CIL public API is also included. See https://github.com/ucsdlib/damsmanager/issues/317#issuecomment-508142934 for more info. Thanks.

[dlasusa]

Changed how the portrait scaling was being calculated. Instead of page_height/watermark_height it's now using page_height/watermark_WIDTH (since the watermark is being rotated). Let me know if that's any better. I'm not sure how the watermarking script is getting called into the overall process. @lsitu

[dlasusa]

Also, as a reminder, when I make changes they're committed to the watermark repository (not damsmanager directly)

[lsitu]

@dlasusa Thanks. We just follow the instruction in the watermark repository https://github.com/ucsdlib/watermark that @rstanonik documents to call the Python scripts. So just make the change in https://github.com/ucsdlib/watermark and we need @rstanonik to redeploy it to staging, QA and prod once it's done. Thank you.

[rstanonik]

@lsitu Thanks for the reminder! I even have a Jenkins plan to deploy, but not configured to deploy automatically. I ran the plan, so it should be ready to test.

[lsitu]

Thanks @dlasusa and @rstanonik for the quick fix. @mdpeters Could you test the PDF wartermarking again? Thank you.

[lsitu]

@mcrichlow I've added PR https://github.com/ucsdlib/damsmanager/pull/343 to fix the file use value issue that @mdpeters reported. It's ready for review now. Thanks.

[lsitu]

@mdpeters I think the static "document-service" file use for image watermarking is fixed and it's ready for test now. Also, thanks @dlasusa and @rstanonik for fixing and deploying the Python script for PDF watermarking (vertical scaling of the watermark image). Could you test the VRR watermarking support again? Thank you.

[lsitu]

@gamontoya / @arwenhutt / @GregReser We've got the batch metadata overlay support ready on staging for test now: https://librarytest.ucsd.edu/damsmanager/batchEdit.do:

Just like the Excel InputStream form, we'll select the Excel source metadata from dams_staging and click the Submit button for Preview. If you want to proceed for ingest, uncheck checkbox Preview the converted RDF/XML only, no ingest. and the Batch metadata overlay. option will be checked. Please let me know if you have any questions. Thank you.

[mdpeters]

@lsitu the image watermarking and display is working but now the PDF display is broken (looking at the data view there doesn't seem to be a document-service component). The PDF objects are behaving like image objects, trying to go to a zoom view rather than opening the watermarked pdf. Examples: https://librarytest.ucsd.edu/dc/object/bd88956018 https://librarytest.ucsd.edu/dc/object/bd7223232k

[lsitu]

@mdpeters It looks like the PDFs for object bd88956018 and bd7223232k were ingested as document-source without the watermarking option. Could you try it one more time and make sure that the option for watermarking is selected to see how it goes? Thank you.

[mdpeters]

damslog--768410633.txt @lsitu watermarking option was definitely checked when they were ingested. The following image objects were ingested at the same time and they did receive a watermark: https://librarytest.ucsd.edu/dc/object/bd65406355 https://librarytest.ucsd.edu/dc/object/bd8247134w

I've attached the damslog for that ingest

[lsitu]

@mdpeters Thanks for the additional information. I am looking into the PDF watermarking issue now. But what's the difference with your ingest test last time that got PDF watermarking working? Can you tell me the location of the source metadata file and the files that use for test? Thank you.

[mdpeters]

@lsitu There isn't much of a difference between last time and this time. I even used many of the same files (I added a few objects to do some additional testing but this object: https://librarytest.ucsd.edu/dc/object/bd88956018 has the exact same data and was ingested the exact same way as this object: https://librarytest.ucsd.edu/dc/object/bd47317414 that I did last week)

Objects and metadata are in \lib-storage\digital-staging\book_backlog\VRR_TESTING\

[gamontoya]

@arwenhutt @abbypenn93 @remerjohnson Can you please test this on staging and let us know if it's working as expected?

[remerjohnson]

@gamontoya To confirm, is this to test VRR functionality?

[lsitu]

@remerjohnson I think @mdpeters is testing the VRR functionality at this time. But we need to test all other functions in https://github.com/ucsdlib/damsmanager/issues/339#issue-461103089 as well. Thank you.

[gamontoya]

@remerjohnson As Longshou stated. You really have to test that the batch metadata overlay is working as expected.

[lsitu]

@mdpeters Thanks for the information. I think I've found the problem, which could be caused by my mistake while fixing merge conflicts somehow. @mcritchlow I've created PR https://github.com/ucsdlib/damsmanager/pull/344 to fix it. It's ready for review now. Thank you.

[mcritchlow]

merged, thanks @lsitu :+1:

[lsitu]

Thank @mcritchlow. @mdpeters We've deployed damsmanager to staging, which will fix the PDF watermarking issue that you found yesterday. Could you test it again? Thank you.

[arwenhutt]

@gamontoya an update on testing of batch edit/overlay - I've drafted a testing plan and test booklet. I have a few more tests to add to it, and will be reviewing it with @abbypenn93 and @remerjohnson and then we'll divy up and start testing.

We're going more formal with this testing due to the great responsibility which comes with great (editing) power!

[lsitu]

@arwenhutt With the heavy codes refactoring for batch overlay support, I think the Excel Standard InputStream is affected by the changes. Could you add test for it as well? Thank you.

[gamontoya]

@arwenhutt Thank you! We'll be very patient with this one since it does merit detailed testing.

[arwenhutt]

@lsitu thanks for the heads up. I'll add it to our testing plan.

[arwenhutt]

Link to issue for batch edit functionality testing: #345

[gamontoya]

@arwenhutt @lsitu There is still testing going on for this release, correct?

[lsitu]

@gamontoya I am not sure. Have you finished all your tests for the release, @arwenhut, @remerjohnson, @mdpeters?

[arwenhutt]

@lsitu & @gamontoya yes, we are still testing. We’re still working through all the permutations of functions/properties in ingest. We’ve checked all properties at the object, component, & sub-components, and are working through the different rights/access form options, clr ingest/overlay, validations, file type/file use combinations, etc.

[lsitu]

@gamontoya [Give dams-curator group access to the DAMS Collection Status Report and File Report Tool](The codes for https://github.com/ucsdlib/damsmanager/issues/349) and Improve output of the DAMS Collection Status Report are deployed to staging and it's ready for review now. Thanks.

[arwenhutt]

@lsitu & @gamontoya Abby and I can both access the Collection Status Report and File Report tools in staging :tada: Thank you!

[lsitu]

@gamontoya Yes. Is there anything still need to be tested?

[gamontoya]

@lsitu Never mind. I think @arwenhutt and others are still testing.

[arwenhutt]

@gamontoya yes, still testing the overlay. Making progress though and feeling optimistic based on how lightning quick @lsitu is squasing bugs!

[gamontoya]

@lsitu @arwenhutt You guys are the best! 🏆

[lsitu]

@arwenhutt / @abbypenn93 Are you still reviewing ticket https://github.com/ucsdlib/damsmanager/issues/317 for CIL Harvesting? We'll move the new release forward to prod if you've done with it. Thank you.

[lsitu]

@rstanonik Basing on the feedbacks from @gamontoya, @arwenhutt and @mdpeters, @mcritchlow and I just discussed the CIL issues that @abbypenn93 brought up, and consider it won't be a blocker for releasing damsmanager, we decide to move damsmanager release 2.72 forward to prod. Could you deploy it to prod at you earliest convenience? Note that the new release has dependency on private_config (https://github.com/ucsdlib/private_config/issues/18) and we need to deploy it first. Thank you.

[rstanonik]

Deployed to production this morning. private_config was deployed first.

[gamontoya]

Kamino cloned this issue to ucsdlib/damsmanager