search

WebCuratorTool / webcurator

The root of the webcurator tool project, containing all modules needed to run a fully functional webcurator tool.
Apache License 2.0
4 stars 1 forks source link

V3.0/file transfer #11

Closed leefrank9527 closed 4 years ago

leefrank9527 commented 4 years ago

Added remote file transfering mechanism for distributed deployment and resolved some issues of gradle script:

  1. Harvest Agent (H1 & H3) --> Store Component: 1) It's configurable to use "copy" mode or "stream" mode, and taking "copy" as the default mode. 2) For "copy" mode, it's the existing method. When Heitrix and Store Component are deployed at the same machine, files can be copyed from none directory to another. 3) For "stream" mode, it's the new added method. When Heitrix and Store Component are deployed at the different machines, files can be transfered via HTTP API. 4) Got rid of the unused "save" APIs to simplify Store Component.
  2. WebApp --> Store Component: When reading imported file from "uploadedFiles" folder at Store Component side, Store Component will check does the file exist. If the file does not exist, it will download the file from WebApp via HTTP API.
  3. Update build.gradle: 1) Resolved the deprecated warnings: to replace "classifier==''" with " archiveClassifier.set('')". 2) Fixed file can't be found error for webapp/build.gradle of updateVersion function.
leefrank9527 commented 4 years ago

@obrienben the two changes above about mysql should be ignored. I'm using Mysql 8 in my developemnt environment, and I didn't test that with Mysql 5.7.

obrienben commented 4 years ago

@leefrank9527 does this PR include the changes for H1 agent? There's no digitalAssetStore.fileUploadMode property in the H1 config files.

leefrank9527 commented 4 years ago

H1 Agent should be included. The missing configuration was added, and was updated to the same branch: v3.0/file-transfer.

obrienben commented 4 years ago

Merged into master through the nlnz-uat-deploy branch as there were several merge conflicts and a couple of changes in this PR to revert.