Delete deprecated file storage directories and files

steinmb commented 7 years ago

Follow up issue from #321 that move more the 4000 files around into standard storage area. It is not a silver bullet and some left overs need human interaction. Issues like these clean up the installation and would also ease when the site get archived.

Left from Drupal 6 and have not been in use since. (computer generated versions)
Left files from old and partly broken configs. Happen before we moved config into code and took control.
files manually copied into the file area. We call them untracked files. Older ways of doing media like IMCE was able to scan and use/attach media that is untracked. This is no longer supported and these files are impossible for us to use.

Files and directories to remove

[x] Partly broken person/author images public://author_photos/[image]
[x] public://files/attachments
[x] public://media/critical-writing moved to 'public://media/critical_writing'
[x] public://media/ news moved to story - Files still pending.
[x] public://files/attachments - (files/attachments)
[x] public://media-youtube_old
[x] public://file_attach
[x] public://file-styles
[ ] public://logos/*
[x] public://imagefield_thumbs/*
[x] public://files/*
[x] public://u/

How to test

Nothing.

Files in Drupal public:// root directory to check

These needs a closer look:

[ ] nobodyAtBap121105.mov
[ ] cell_workshop_programme_final.pdf
[x] creative_commons_attribution-sharealike_3.png
[ ] rts-cover-front-page-web.jpg
[ ] DemonOfAnalogySm.mov
[ ] rts_schedule.pdf
[ ] loss-undersea-video.mp4
[ ] 1207-2004-04-29-1-Carpenter.mp3
[ ] rts-cover-front-page-web_675x597.jpg
[ ] jrcarpenter.taroko.mov
[ ] hollan_teaser.mp4
[x] elmcip_favicon.png
[ ] elcmip_presentation_slide.png
[ ] poster_kopie.jpg
[ ] maxresdefault.jpg
[ ] vso_programa_rj_final.pdf
[ ] img_0251.mov
[ ] alanbigelow.reading.mov
[ ] newsWheel.mov
[ ] eca-logo_70x70.png
[x] elmcip_favicon_0.png
[ ] img_0251_0.mov
[x] _logo_0.png
[ ] ormstad1.png
[ ] cell-workshop1.jpg
[ ] if-I-wrote-you-a-love-letter-video.mp4
[ ] mir.doc
[x] _logo_1.png
[ ] RW_tiny.mp4
[ ] img_5335.jpg
[ ] img_5337.jpg
[ ] post-me-moires.avi
[ ] eca_uoe_75x310.png
[ ] eca_logo_75x75.png
[x] pixture_reloaded_logo.png
[ ] img_5328.jpg
[ ] cell-workshop2.jpg
[x] _logo.png

steinmb commented 7 years ago

These seems partly broken. public://author_photos/[image]

sites/default/files/author_photos/
-rwxrwxr-x. 1 steinmb ns9035k     0 Feb 26  2013 alanliu2002-thumb.jpg
-rwxrwxr-x. 1 steinmb ns9035k 26045 Feb 26  2013 eric_rasmussen_chicago.jpg
-rwxrwxr-x. 1 steinmb ns9035k 26732 Feb 26  2013 john_cayley.jpg
-rwxrwxr-x. 1 steinmb ns9035k 10969 Feb 26  2013 judy_bio_0.jpg
-rwxrwxr-x. 1 steinmb ns9035k 92767 Feb 26  2013 skype-bild.png

SELECT file_managed.fid, uri FROM file_managed INNER JOIN file_usage ON file_managed.fid = file_usage.fid WHERE uri LIKE 'public://author_photos/%';
+------+-----------------------------------------------------------------------------------+
| fid  | uri                                                                               |
+------+-----------------------------------------------------------------------------------+
| 1339 | public://author_photos/1423472.jpeg                                               |
|  951 | public://author_photos/148483_114172111981776_100001669642566_93534_7195447_n.jpg |
| 1356 | public://author_photos/2004-lifewires-krokers.png                                 |
| 1527 | public://author_photos/87aaas_moretti.jpg                                         |
|  896 | public://author_photos/alanliu2002-thumb.jpg                                      |
|  897 | public://author_photos/alanliu2002-thumb_0.jpg                                    |
|  979 | public://author_photos/beat_suter_360.png                                         |
| 1519 | public://author_photos/ciccoricco11.jpg                                           |
|  841 | public://author_photos/eric_rasmussen_chicago.jpg                                 |
| 1148 | public://author_photos/fabio_de_vivo.jpg                                          |
| 1618 | public://author_photos/facetofacebook_alessandro-ludovico_portrait.jpg            |
| 1194 | public://author_photos/images-2.jpeg                                              |
|  943 | public://author_photos/img_1202-264x300.jpg                                       |
|  944 | public://author_photos/ismar_img.jpg                                              |
| 1323 | public://author_photos/jb.pasfoto.2006.jpg                                        |
|  865 | public://author_photos/johannes_helden2_-1.jpg                                    |
|  895 | public://author_photos/john_cayley.jpg                                            |
| 1613 | public://author_photos/jrc_map2012.jpg                                            |
|  981 | public://author_photos/judy_bio_0.jpg                                             |
|  863 | public://author_photos/l1050393.jpg                                               |
| 1208 | public://author_photos/large_melissa.lucas_.jpg                                   |
| 1336 | public://author_photos/nm_self_portrait.jpg                                       |
|  862 | public://author_photos/photo_cv_luc.jpg                                           |
| 1354 | public://author_photos/skype-bild.png                                             |
|  891 | public://author_photos/teo_faca_za_elmcip.jpg                                     |
| 1355 | public://author_photos/unknown.jpeg                                               |
| 1472 | public://author_photos/unknown_0.jpeg                                             |
| 1512 | public://author_photos/unknown_1.jpeg                                             |
| 1308 | public://author_photos/white_knight.jpg                                           |
+------+-----------------------------------------------------------------------------------+
29 rows in set (0.01 sec)

steinmb commented 7 years ago

Old attachments to look into

|   1986 | public://files/attachments/criticalwriting/a_response_to_nick_montfort.pdf                                                                                       |
|   1856 | public://files/attachments/criticalwriting/blended-learning-transatlantic-teaching_tomaszek.pdf                                                                  |
|   2094 | public://files/attachments/criticalwriting/collective_memory_and_the_development_of_a_field_cut.pdf                                                              |
|   2098 | public://files/attachments/criticalwriting/collective_memory_and_the_development_of_a_field_cut_0.pdf                                                            |
|  42831 | public://files/attachments/criticalwriting/elofinaldraft.pdf                                                                                                     |
|   1234 | public://files/attachments/criticalwriting/generatingbooks_jrcarpenter.pdf                                                                                       |
|   1913 | public://files/attachments/criticalwriting/interpoesie_em_frances_final.pdf                                                                                      |
|   1706 | public://files/attachments/criticalwriting/nick_montford-programming_for_fun.rtf                                                                                 |
|  42779 | public://files/attachments/criticalwriting/rettberg_baldwin_elmcip.pdf                                                                                           |
|   1482 | public://files/attachments/criticalwriting/tomasula_toc.pdf                                                                                                      |
|   1795 | public://files/attachments/criticalwriting/tomaszek_master-thesis-synopsis.pdf                                                                                   |
|  42686 | public://files/attachments/organization/elrg_report_2013_optimized.pdf                                                                                           |
|  42492 | public://files/attachments/person/2013-04-24_cv.pdf                                                                                                              |
| 223582 | public://files/attachments/person/cv.txt                                                                                                                         |
| 306615 | public://files/attachments/person/cv_2.pdf                                                                                                                       |
|  42669 | public://files/attachments/person/gattasscvmay13.pdf                                                                                                             |
| 273792 | public://files/attachments/person/lsyr_llmy_wldhty_lldktwr_yd.pdf                                                                                                |
|   1206 | public://files/attachments/story/digital_poetics_and_the_present_seminarpdf.pdf                                                                                  |
|   1197 | public://files/attachments/story/digital_poetics_program_and_the_present.pdf                                                                                     |
|   1001 | public://files/attachments/work/wilks-submission_0.txt

sites/default/files/attachments/criticalwriting:

-rwxrwxr-x. 1 steinmb ns9035k  7291420 Feb 26  2013 2003_multimedia_criticism_us_letter.pdf
-rwxrwxr-x. 1 steinmb ns9035k 25262982 Feb 26  2013 bergen_01_serge_bouchardon.mp3
-rwxrwxr-x. 1 steinmb ns9035k 23606405 Feb 26  2013 bergen_06_nick_montfort.mp3
-rwxrwxr-x. 1 steinmb ns9035k 36839448 Feb 26  2013 bergen_06_yra_vandijk.mp3
-rwxrwxr-x. 1 steinmb ns9035k 23462979 Feb 26  2013 bergen_07_andrew_roberts.mp3
-rwxrwxr-x. 1 steinmb ns9035k 15499056 Feb 26  2013 bergen_08_jill_walker_rettberg.mp3
-rwxrwxr-x. 1 steinmb ns9035k 22531344 Feb 26  2013 bergen_09_scott-rettberg.mp3
-rwxrwxr-x. 1 steinmb ns9035k 21254490 Feb 26  2013 bergen_15_raine_koskimaa.mp3
-rwxrwxr-x. 1 steinmb ns9035k 21722781 Feb 26  2013 bergen_16_giovanna_di_rosario.mp3
-rwxrwxr-x. 1 steinmb ns9035k 25745997 Feb 26  2013 bergen_17_rob_wittig.mp3
-rwxrwxr-x. 1 steinmb ns9035k 15499056 Feb 26  2013 bergen_18_jill_walker_rettberg.mp3
-rwxrwxr-x. 1 steinmb ns9035k 22531344 Feb 26  2013 bergen_19_scott-rettberg.mp3
-rwxrwxr-x. 1 steinmb ns9035k   163726 Feb 26  2013 husarova_fullpaper.pdf
-rwxrwxr-x. 1 steinmb ns9035k   253750 Feb 26  2013 tomaszek_fullpaper.pdf

sites/default/files/attachments/news:
-rwxrwxr-x. 1 steinmb ns9035k 1191180 Feb 26  2013 elmcip_ec_seminar_program_final_print.pdf
-rwxrwxr-x. 1 steinmb ns9035k 2490368 Feb 26  2013 ELMCIP presentation.ppt

sites/default/files/attachments/work:
-rwxrwxr-x. 1 steinmb ns9035k 29997259 Feb 26  2013 creeleymix-2.zip
-rwxrwxr-x. 1 steinmb ns9035k   111245 Feb 26  2013 fittingthepattern_screenshot.png
-rwxrwxr-x. 1 steinmb ns9035k        0 Feb 26  2013 screen_shot_2011-01-05_at_13.29.12.png

steinmb commented 7 years ago

critical-writing vs critical_writing. These have to be moved updated to critical_writing.

SELECT file_managed.fid, filename, uri FROM file_managed INNER JOIN file_usage ON file_managed.fid = file_usage.fid WHERE uri like 'public://media/critical-writing/%';
+--------+----------------------------------+-------------------------------------------------------------------------+
| fid    | filename                         | uri                                                                     |
+--------+----------------------------------+-------------------------------------------------------------------------+
|    401 | 320.jpeg                         | public://media/critical-writing/images/320.jpeg                         |
|   4179 | a-vilaghalo-metaforai.jpg        | public://media/critical-writing/images/a-vilaghalo-metaforai.jpg        |
|  42827 | cover_rubisttxt.jpg              | public://media/critical-writing/images/cover_rubisttxt.jpg              |
|    195 | elc_1_screen_shot.png            | public://media/critical-writing/images/elc_1_screen_shot.png            |
|    197 | elc_1_screen_shot.png            | public://media/critical-writing/images/elc_1_screen_shot_1.png          |
|    198 | elc_screen_shot.png              | public://media/critical-writing/images/elc_screen_shot.png              |
|   4182 | griotsings.png                   | public://media/critical-writing/images/griotsings.png                   |
|   1357 | i_love_e-poetry.png              | public://media/critical-writing/images/i_love_e-poetry.png              |
|   2713 | iloveepoetryscreencapture.png    | public://media/critical-writing/images/iloveepoetryscreencapture.png    |
| 307023 | image.jpeg                       | public://media/critical-writing/images/image.jpeg                       |
|   2297 | paratextual-web-architecture.png | public://media/critical-writing/images/paratextual-web-architecture.png |
|  42920 | pdolspace.gif                    | public://media/critical-writing/images/pdolspace.gif                    |
|  42919 | smart_p.gif                      | public://media/critical-writing/images/smart_p.gif                      |
|   2296 | the-web-is-paratextual.png       | public://media/critical-writing/images/the-web-is-paratextual.png       |
|   1659 | tokyo_garage.jpg                 | public://media/critical-writing/images/tokyo_garage.jpg                 |
| 307544 | torres_uc_31032017_img_9688.jpeg | public://media/critical-writing/images/torres_uc_31032017_img_9688.jpeg |
|   2288 | valliasblocknavas.png            | public://media/critical-writing/images/valliasblocknavas.png            |
+--------+----------------------------------+-------------------------------------------------------------------------+

steinmb commented 7 years ago

Cleaning up the above. Exposed data duplication.

Source: public://media/critical-writing/images/elc_1_screen_shot.png
New: public://media/critical_writing/images/elc_1_screen_shot.png
Source: public://media/critical-writing/images/elc_1_screen_shot_1.png
New: public://media/critical_writing/images/elc_1_screen_shot_1.png
Source: public://media/critical-writing/images/elc_screen_shot.png
New: public://media/critical_writing/images/elc_screen_shot.png
Source: public://media/critical-writing/images/320.jpeg
New: public://media/critical_writing/images/320.jpeg
Source: public://media/critical-writing/images/i_love_e-poetry.png
New: public://media/critical_writing/images/i_love_e-poetry.png
Source: public://media/critical-writing/images/tokyo_garage.jpg
New: public://media/critical_writing/images/tokyo_garage.jpg
Source: public://media/critical-writing/images/valliasblocknavas.png
New: public://media/critical_writing/images/valliasblocknavas.png
Source: public://media/critical-writing/images/the-web-is-paratextual.png
New: public://media/critical_writing/images/the-web-is-paratextual.png
Source: public://media/critical-writing/images/paratextual-web-architecture.png
New: public://media/critical_writing/images/paratextual-web-architecture.png
Source: public://media/critical-writing/images/iloveepoetryscreencapture.png
New: public://media/critical_writing/images/iloveepoetryscreencapture.png
Source: public://media/critical-writing/images/a-vilaghalo-metaforai.jpg
New: public://media/critical_writing/images/a-vilaghalo-metaforai.jpg
Source: public://media/critical-writing/images/griotsings.png
New: public://media/critical_writing/images/griotsings.png
Source: public://media/critical-writing/images/cover_rubisttxt.jpg
New: public://media/critical_writing/images/cover_rubisttxt.jpg
Source: public://media/critical-writing/images/smart_p.gif
New: public://media/critical_writing/images/smart_p.gif
Source: public://media/critical-writing/images/pdolspace.gif
New: public://media/critical_writing/images/pdolspace.gif
Source: public://media/critical-writing/images/uqamphoto_22sept2014.jpg
New: public://media/critical_writing/images/uqamphoto_22sept2014.jpg
Source: public://media/critical-writing/images/uqamposter_22sept2014.jpg
New: public://media/critical_writing/images/uqamposter_22sept2014.jpg
Source: public://media/critical-writing/images/image.jpeg
New: public://media/critical_writing/images/image.jpeg
Source: public://media/critical-writing/images/torres_uc_31032017_img_9688.jpeg
New: public://media/critical_writing/images/torres_uc_31032017_img_9688.jpeg
-bash-4.1$
WD php: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://media/critical_writing/images/tokyo_garage.jpg' for key 'uri':     [error]
UPDATE {file_managed} SET uid=:db_update_placeholder_0, filename=:db_update_placeholder_1, uri=:db_update_placeholder_2, filemime=:db_update_placeholder_3,
filesize=:db_update_placeholder_4, status=:db_update_placeholder_5, timestamp=:db_update_placeholder_6, type=:db_update_placeholder_7
WHERE  (fid = :db_condition_placeholder_0) ; Array
(
    [:db_update_placeholder_0] => 35
    [:db_update_placeholder_1] => tokyo_garage.jpg
    [:db_update_placeholder_2] => public://media/critical_writing/images/tokyo_garage.jpg
    [:db_update_placeholder_3] => image/jpeg
    [:db_update_placeholder_4] => 172916
    [:db_update_placeholder_5] => 1
    [:db_update_placeholder_6] => 1498130621
    [:db_update_placeholder_7] => image
    [:db_condition_placeholder_0] => 1659
)
 in drupal_write_record() (line 7383 of /applications/elmcip.net/drupal/includes/common.inc).
Cannot modify header information - headers already sent by (output started at /applications/elmcip.net/bin/person.php:16) bootstrap.inc:1486                             [warning]
PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry &#039;public://media/critical_writing/images/tokyo_garage.jpg&#039; for key &#039;uri&#039;: UPDATE {file_managed} SET uid=:db_update_placeholder_0, filename=:db_update_placeholder_1, uri=:db_update_placeholder_2, filemime=:db_update_placeholder_3, filesize=:db_update_placeholder_4, status=:db_update_placeholder_5, timestamp=:db_update_placeholder_6, type=:db_update_placeholder_7
WHERE  (fid = :db_condition_placeholder_0) ; Array
(
    [:db_update_placeholder_0] => 35
    [:db_update_placeholder_1] => tokyo_garage.jpg
    [:db_update_placeholder_2] => public://media/critical_writing/images/tokyo_garage.jpg
    [:db_update_placeholder_3] => image/jpeg
    [:db_update_placeholder_4] => 172916
    [:db_update_placeholder_5] => 1
    [:db_update_placeholder_6] => 1498130621
    [:db_update_placeholder_7] => image
    [:db_condition_placeholder_0] => 1659
)
 in drupal_write_record() (line 7383 of /applications/elmcip.net/drupal/includes/common.inc).

+------+------------------+---------------------------------------------------------+
| fid  | filename         | uri                                                     |
+------+------------------+---------------------------------------------------------+
| 1659 | tokyo_garage.jpg | public://media/critical-writing/images/tokyo_garage.jpg |
| 1660 | tokyo_garage.jpg | public://media/critical_writing/images/tokyo_garage.jpg |
+------+------------------+---------------------------------------------------------+

Removed https://elmcip.net/file/1659 - https://elmcip.net/file/1660 both pointed to identical file.

steinmb commented 7 years ago

Doh, another one:

SELECT file_managed.fid, filename, uri FROM file_managed INNER JOIN file_usage ON file_managed.fid = file_usage.fid WHERE uri like 'public://media/%valliasblocknavas.png';
+------+-----------------------+--------------------------------------------------------------+
| fid  | filename              | uri                                                          |
+------+-----------------------+--------------------------------------------------------------+
| 2288 | valliasblocknavas.png | public://media/critical-writing/images/valliasblocknavas.png |
| 2289 | valliasblocknavas.png | public://media/event/attachments/valliasblocknavas.png       |
| 2290 | valliasblocknavas.png | public://media/critical_writing/images/valliasblocknavas.png |
+------+-----------------------+--------------------------------------------------------------+

Used in:

Removed file entity https://elmcip.net/file/2288/

steinmb commented 7 years ago

Aaaand a few more that I manually needed to address.

+------+----------------------------+-------------------------------------------------------------------+
| fid  | filename                   | uri                                                               |
+------+----------------------------+-------------------------------------------------------------------+
| 2296 | the-web-is-paratextual.png | public://media/critical-writing/images/the-web-is-paratextual.png |
| 2298 | the-web-is-paratextual.png | public://media/critical_writing/images/the-web-is-paratextual.png |
+------+----------------------------+-------------------------------------------------------------------+

+------+----------------------------------+-------------------------------------------------------------------------+
| fid  | filename                         | uri                                                                     |
+------+----------------------------------+-------------------------------------------------------------------------+
| 2297 | paratextual-web-architecture.png | public://media/critical-writing/images/paratextual-web-architecture.png |
| 2300 | paratextual-web-architecture.png | public://media/critical_writing/images/paratextual-web-architecture.png |
+------+----------------------------------+-------------------------------------------------------------------------+

+--------+------------+---------------------------------------------------+
| fid    | filename   | uri                                               |
+--------+------------+---------------------------------------------------+
| 307023 | image.jpeg | public://media/critical-writing/images/image.jpeg |
| 307210 | image.jpeg | public://media/critical_writing/images/image.jpeg |
+--------+------------+---------------------------------------------------+

+--------+----------------------------------+-------------------------------------------------------------------------+
| fid    | filename                         | uri                                                                     |
+--------+----------------------------------+-------------------------------------------------------------------------+
| 307544 | torres_uc_31032017_img_9688.jpeg | public://media/critical-writing/images/torres_uc_31032017_img_9688.jpeg |
| 307545 | torres_uc_31032017_img_9688.jpeg | public://media/critical_writing/images/torres_uc_31032017_img_9688.jpeg |
+--------+----------------------------------+-------------------------------------------------------------------------+

steinmb commented 7 years ago

Cleaned out quite a few. Here is one fun issue:

SELECT file_managed.fid, uri FROM file_managed INNER JOIN file_usage ON file_managed.fid = file_usage.fid WHERE uri LIKE 'public://files/%';
+--------+-------------------------------------------------------------------------------------------------------+
| fid    | uri                                                                                                   |
+--------+-------------------------------------------------------------------------------------------------------+
|   1986 | public://files/attachments/criticalwriting/a_response_to_nick_montfort.pdf                            |
|   1856 | public://files/attachments/criticalwriting/blended-learning-transatlantic-teaching_tomaszek.pdf       |
|   2094 | public://files/attachments/criticalwriting/collective_memory_and_the_development_of_a_field_cut.pdf   |
|   2098 | public://files/attachments/criticalwriting/collective_memory_and_the_development_of_a_field_cut_0.pdf |
|  42831 | public://files/attachments/criticalwriting/elofinaldraft.pdf                                          |
|   1234 | public://files/attachments/criticalwriting/generatingbooks_jrcarpenter.pdf                            |
|   1913 | public://files/attachments/criticalwriting/interpoesie_em_frances_final.pdf                           |
|   1706 | public://files/attachments/criticalwriting/nick_montford-programming_for_fun.rtf                      |
|  42779 | public://files/attachments/criticalwriting/rettberg_baldwin_elmcip.pdf                                |
|   1482 | public://files/attachments/criticalwriting/tomasula_toc.pdf                                           |
|   1795 | public://files/attachments/criticalwriting/tomaszek_master-thesis-synopsis.pdf                        |
|  42686 | public://files/attachments/organization/elrg_report_2013_optimized.pdf                                |
|  42492 | public://files/attachments/person/2013-04-24_cv.pdf                                                   |
| 223582 | public://files/attachments/person/cv.txt                                                              |
| 306615 | public://files/attachments/person/cv_2.pdf                                                            |
|  42669 | public://files/attachments/person/gattasscvmay13.pdf                                                  |
| 273792 | public://files/attachments/person/lsyr_llmy_wldhty_lldktwr_yd.pdf                                     |
|   1206 | public://files/attachments/story/digital_poetics_and_the_present_seminarpdf.pdf                       |
|   1197 | public://files/attachments/story/digital_poetics_program_and_the_present.pdf                          |
|   1001 | public://files/attachments/work/wilks-submission_0.txt                                                |
+--------+-------------------------------------------------------------------------------------------------------+

steinmb commented 7 years ago

Only post release work left.

steinmb commented 7 years ago

Most stuff cleaned up. Other can pick this is if they feel like it.

Center-for-Digital-Narrative / elmcip