Migrate files

[cbrody]

Move files in root of public files directory into dated subdirectories, e.g. files/yyyy/mm/dd/filename.jpg

[cbrody]

Drush script is ready for testing. Prior to running on production site we need to modify all content types to save new images, files, pdfs, etc. into dated subdirectories. We can use tokens such as: /[current-date:custom:Y/m/d]

[cbrody]

Usage: drush mf --file_type=[img|doc|pdf] --source_dir=<source> --target-dir=<target> e.g.

drush mf --file_type=img --target_dir=images
drush mf --file_type=doc --target_dir=docs
drush mf --file_type=pdf --target_dir=pdfs
drush mf --file_type=img --source_dir=images --target_dir=images

[cbrody]

All blocks using custom image paths (i.e. images inserted not using an image field) will need to be updated manually, e.g. on the /news-comment page.

[cbrody]

Re image paths in body fields, we're reviewing which content types apart from blogs might be affected. @finnlewis could you look at extending the drush script to re-write <img> src attributes to be relative links to the new, date-based file locations?

[finnlewis]

Thanks @cbrody , I'll try to take a look at this this week.

[finnlewis]

Hi @cbrody (forgot to post this yesterday)

I've made some progress on this, please see the following commit:

https://github.com/IIED-org/iiedorg/commit/6fadfacde8c05a0a7b1be40129215c83da4abf91

I've run it locally and I think it does what we need. It took over an hour to run though!

Happy to review together and iterate on it if needed.

Cheers,

Finn

[cbrody]

@finnlewis could you let me know when this is ready for testing with the updates we discussed please?

[finnlewis]

Hi @cbrody

Just looking at this one now.

[finnlewis]

Hi @cbrody

I've updated the code on this and created a pull request for our review.

Just refreshing the database and files from the live to test again, which will take a while so I'll do something else while that refreshes.

Here's the pull request https://github.com/IIED-org/iiedorg/pull/118/files

[finnlewis]

I made an interesting mistake on this. I didn't specify the filename to update when updating links to https//www.iied.org/sites/default/files and http://www.iied.sites/default/files....

BUT this might be just fine.... as on the first run through the loop it replaces all links in each table.

So the first output from running lando drush mf is as follows:

Replaced 10 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in block_custom
Replaced 287 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_body
Replaced 1 link(s) to http://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_body
Replaced 3 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_additional_resources
Replaced 1 link(s) to http://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_additional_resources
Replaced 51 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_basic_text
Replaced 7 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_contact_0
Replaced 29 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_downloads
Replaced 1 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_funded_by
Replaced 1 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_glossary
Replaced 1 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_howtoapply
Replaced 3 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_links
Replaced 2 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_overlay_text
Replaced 2 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_series_badge
Replaced 6 link(s) to https://www.iied.org/sites/default/files/ with /sites/default/files/ in field_data_field_source

Then on subsequent loops there is nothing to replace.

[finnlewis]

Hi @cbrody

Can I help you to test and run this?

[finnlewis]

@cbrody will test and let me know if there are any issues.

[cbrody]

Random list of pages to check post file migration: /iieds-best-2021-blogs /iied-50-making-iied-future-fit /iied-50-reflecting-past-looking-future-make-change-happen-podcast-episode-15 /iieds-best-2021-publications /multiple-housing-climate-related-vulnerabilities-informal-workers-indore-india /ugandan-diet-food-voices-choices /information-value-chains-data-research-communication /mining-cobalt-better /how-communities-are-protecting-their-biocultural-resources-community-protocols

Galleries: /image-gallery-2014-barbara-ward-lecture /connecting-settlements-cities-basins-realising-sdg-6-scale /pictures-dialogue-for-action-mining-ghana /photo-exhibition-highlights-varied-vibrant-informal-economies /food-waste-can-grassroots-initiatives-stop-us-throwing-good-food-bin

Landing pages: /about /news-comment /sign-up /iied-online /our-other-sites-networks

Strategy: /iied-strategy-2019-2024

Pages with PDFs e.g. /responsible-operations-2020

@ncktrnr can you add any others that we should check please?

[cbrody]

Hi @finnlewis lines 165 and 188 in the script look wrong to me - are they missing a forward slash prior to $target_directory_for_file ? The output of the command (still running at time of writing) looks like this:

72%
Replaced 1 links to styles file path with publicimages/2018/06/26/2018-06-26_tanzaniaenergygraph-169.png

[finnlewis]

Hi @cbrody

Good catch, I think I was testing without specifying a --target_dir=images for example, in which case the $target_directory_for_file variable starts with a backslash.

I'm just amending the script to add those slashes on lines 165 and 188, but also to trim off the initial backslash if present.

[cbrody]

I've also made some changes to the script as we need to update paths for files in the root /files directory, so I'll merge your changes into mine when ready.

[ncktrnr]

@ncktrnr can you add any others that we should check please?

Here's a list of the 12 nodes using file upload paragraph https://www.iied.org/admin/file-uploads?bundle=file_upload The view should be useful to check a few nodes using image paragraphs, although these are mostly restricted to long reads

[cbrody]

Thanks @ncktrnr, very useful. There is an issue with a long read (/mist-clears-china-african-forests) where the image paths are coded as src="/files/<filename.ext>" but this is the only one like that as far as I can see...

[cbrody]

Please see attached file testpaths.txt for sample paths to check post clean-up.

[cbrody]

Closing this as process completed successfully on stage and pushed to production.