Mnylc issue93

[patdunlavey]

Ticket 93

What does this Pull Request do?

• When fetching remote files, the root of the public:// files directory can become filled with very large files, necessitating massive storage where it may not be appropriate.
• Temporary files and metadata for batch ingests should not be stored in a location that is freely open to the public.

What's new?

• Makes the temporary file storage location configurable.
• Puts temporary file storage under the private:// directory, rather than public://

How should this be tested?

• Prepare and process an IMI ingest which uses Amazon S3 for file source. The uploaded csv file should be found in private://islandora-multi-importer/csv. The ingest should be successful, and the fetched files should be saved into private://islandora-multi-importer.
• Go to the imi settings page (/admin/islandora/tools/multi_importer) and change the "Temporary files storage directory". When running a new ingest, the directory should get created, and the csv and downloaded files should be saved to folders within this directory.
• When deleting an ingest batch, the temporary files should be removed from wherever they were stored, even if the storage location configuration changed.

Additional Notes:

• Does this change require documentation to be updated? Yes, the documentation for the IMI settings page should be updated.
• Does this change add any new dependencies? No
• Does this change require any other modifications to be made to the repository (ie. Regeneration activity, etc.)? No.
• Could this change impact execution of existing code? I don't think so.

Interested parties

@DiegoPino

[DiegoPino]

Ok!! i support you on that... but we do allow people to use public if they want? Just asking, not a blocker. Can test tomorrow and once passing basic battery of handcrafted use cases can merge!

El El mar, 20 de nov. de 2018 a las 19:14, Pat Dunlavey < notifications@github.com> escribió:

@patdunlavey commented on this pull request.

In includes/admin.form.inc https://github.com/mnylc/islandora_multi_importer/pull/94#discussion_r235215617 :

@@ -41,7 +41,12 @@ function islandora_multi_importer_admin_form($form, $form_state) { '#description' => t('This option also requires corresponding Drupal permissions to be correctly set.'), '#default_value' => variable_get('islandora_multi_importer_extrads', FALSE), );

• $form['islandora_multi_importer_generalsettings_fieldset']['islandora_multi_importer_temp_dir'] = array(
• '#type' => 'textfield',
• '#title' => t('Temporary files storage directory'),
• '#description' => t('Path to where to store temporary files under Drupal\'s private:// directory. This is used for storing batch ingest files, generated metadata files, and other temporary files. Do not include leading or trailing slash. The private:// directory location is configured at Configuration > Media > File system' ),
• '#default_value' => variable_get('islandora_multi_importer_temp_dir', 'islandora-multi-importer'),
• );

Woodshedding done. It now checks to see if the "private" stream wrapper is defined. If not, it will not save the temporary file directory and throw a validation error when you try. I decided not to failover to "public".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/pull/94#discussion_r235215617, or mute the thread https://github.com/notifications/unsubscribe-auth/AGn8501EgE3MxPViVbPPIWG1Xguc3IPWks5uxJrNgaJpZM4YrQVw .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

[DiegoPino]

Mmm... i will prob do tomorrow. Not trusting that part. Library but lets discuss this. Prob easier to validate the path on form save and make sure it has no slash at the beginning? Let me sleep over this

El El mar, 20 de nov. de 2018 a las 19:21, Pat Dunlavey < notifications@github.com> escribió:

@patdunlavey commented on this pull request.

In composer.json https://github.com/mnylc/islandora_multi_importer/pull/94#discussion_r235216929 :

@@ -8,7 +8,8 @@ "react/http": "^0.4.1", "phpoffice/phpexcel": "^1.8", "google/apiclient": "^2.0",

• "cache/filesystem-adapter": "^0.3.3"
• "cache/filesystem-adapter": "^0.3.3",
• "jakeasmith/http_build_url": "^1"

I could be talked out of this...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnylc/islandora_multi_importer/pull/94#pullrequestreview-177032660, or mute the thread https://github.com/notifications/unsubscribe-auth/AGn850KSNCdp_qKQ7t1PvrilLsqP333Lks5uxJyTgaJpZM4YrQVw .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

[DiegoPino]

@patdunlavey sorry for letting this stale for so long. Could you quickly rebase this to get it merged ASAP? Thanks a lot, there is a lot of good and hard work into this pull and community will for sure appreciate the improvement

[patdunlavey]

I'll try to turn this around today.

[patdunlavey]

Gone stale. Closing.