StationX__NA__NA ghost files in renamed output

[KatieTjaden-McClement]

@ChrisBeirne Some stations had extra un-openable files in the output additional to the actual images. I have attached one of these ghost files.

[lastew]

I had the same thing with Tuyeta - these seem to be a camera failure. I've attached an image of the un-named files, they are already ghosts so the image renaming script isn't losing any information, except the place they are in the sequence. At Tuyeta stations I checked the original files to see where they were, see if it's corresponding to a time when the camera was maybe out of service (we had lots of ghost images where there were supposed to be time lapse images), then just delete them.

[KatieTjaden-McClement]

Just had this happen again, but slightly differently. Laura I believe this is a slightly different issue, as the additional NANA file isn't coming from an actual image since they're all there properly renamed. It seems to be coming from this "Thumbs" file that shows up after copying the directory.

[ChrisBeirne]

We had some failed image writes in NWT, they produced the NA’s... I think the thumbs file is a Mac thing maybe?

Sent from my iPhone

On May 20, 2021, at 1:22 PM, KatieTjaden-McClement @.***> wrote:

 Just had this happen again, but slightly differently. Laura I believe this is a slightly different issue, as the additional NANA file isn't coming from an actual image since they're all there properly renamed. It seems to be coming from this "Thumbs" file.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[KatieTjaden-McClement]

I've done this all on windows, here's some info I found on thumbs files (although I can't change the settings as suggested here they seem to be out of date): https://kb.iu.edu/d/anha. Would there be a simple way to only copy jpg file types or exclude .db files?

[ChrisBeirne]

Yep! Should be able to exclude them from the file list fairly easily... I’ll have a look now

Sent from my iPhone

On May 20, 2021, at 2:30 PM, KatieTjaden-McClement @.***> wrote:

 I've done this all on windows, here's some info I found on thumbs files (although I can't change the settings as suggested here they seem to be out of date): https://kb.iu.edu/d/anha. Would there be a simple way to only copy jpg file types or exclude .db files?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[lastew]

I kind of liked having R try to rename the thumb / error files, because in doing so it spat out a warning message which flagged their existence - otherwise, I would have probably never noticed there was a problem.

[ChrisBeirne]

I think it is good for the broken jpegs, but we should probably nuke the db files

On Thu, 20 May 2021 at 14:49, Laura Stewart @.***> wrote:

I kind of liked having R try to rename the thumb / error files, because in doing so it spat out a warning message which flagged their existence - otherwise, I would have probably never noticed there was a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/WildCoLab/ImageRenamer/issues/2#issuecomment-845500318, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEJVOBZRRR3ZQ77GBJSD5DTOV7XNANCNFSM44FSC2BQ .

[lastew]

that's fair

[mitch-fen]

I think it is good for the broken jpegs, but we should probably nuke the db files …

Agreed!

[ChrisBeirne]

If you pull the code again - it should kill the .db files. If you find any more problem extensions let me know.

Love that we are all in on GitHub team!

Also - this image renamer has got pretty long and unwieldy, we might want to streamline it at some point in the future!

On Thu, 20 May 2021 at 15:00, Mitch Fennell @.***> wrote:

I think it is good for the broken jpegs, but we should probably nuke the db files … <#m2032966991508386875>

Agreed!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/WildCoLab/ImageRenamer/issues/2#issuecomment-845505513, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEJVODWRM5FKGOJFH3VLLTTOWBBTANCNFSM44FSC2BQ .

[mitch-fen]

If you pull the code again - it should kill the .db files. If you find any more problem extensions let me know. Love that we are all in on GitHub team! Also - this image renamer has got pretty long and unwieldy, we might want to streamline it at some point in the future! …

Good call! Maybe we can make it a DM meeting topic to practice optimization!

[liambren01]

Hey all, I ran into some similar issues when processing some of Maddy's goat data. There was the odd camera check with these weird .JPG.thb files (we assume created by a weird card reader). I only noticed there was any issues at the face blur step because the ghost jpegs created by megadector were unreadable. Megadector also treated these files are real images so it created false json entries as well. At this stage because megadector took 12+ hrs to run, I just wrote code to both delete the ghost jpegs and edit the jsons. Ideally, in the future we resolve this issue before anyone gets to the mega detector step by deleting any thb files at the Renaming step. I think all you would need is to add more code very similar to the db delete step.

See code and image below.

`############################################################################## ​

Modified Wildco Image Re-namer script to fix errors in faceblur process. Specifically, iterate

through folders and delete ghost jpegs in the folders, and open json file and delete erroneous entries

​ ############################################################################### ​ setwd("D:/WildCo_Face_Blur") renamed_location <- "Wrazej_Goats_2023" ​

IMPORTANT

Specify if your images are organised into check/deployment subfolders folders which you want to

preserve in your renamed data e.g RICH01/CheckDate1 and ALGAR01/CheckDate2

keep_structure <- TRUE

NOTE - IF YOU USE THIS THEN ALL STATION FOLDERS MUST HAVE A NESTED CHECK/DEPLOYMENT FOLDER

​

If you want to remove that information and merge the images into one folder, specify FALSE

keep_structure <- FALSE

​ ################################ ​

Create a folder to copy the original images into

dir.create(renamed_location) ​

Copy over the files

copyDirectory(to_be_renamed, renamed_location, private=TRUE, recursive=TRUE) ​

Check the exif tool is working on the original data

test <- list.files(path = renamed_location, full.names = T, include.dirs = F, recursive = T)[1] ​ read_exif(test, tags = c("DateTimeOriginal"), recursive = F, quiet = TRUE) ​

You should see a tibble containing the file names and DateTimeOriginal

If this doesn't work - check your exif installation!

​ ​

Get a list of the folders to be renamed

Folders <- list.dirs(path = renamed_location, full.names = FALSE) ​ ########################################3 ​

Remove the empty argument

Folders <- Folders[Folders!=""] ​

Remove any hollow folders -> folders that don't contain images (e.g. if you are using nested folders, some folders will just contain directories)

for(i in 1: length(Folders)) { tmp <- dir.exists(list.files(path = paste0(renamed_location, "//",Folders[i]), full.names = T, include.dirs = T))

Remove folders that just contain folders

if(length(tmp[tmp==TRUE])>0) { Folders[i] <- "" }

Remove folders that dont contain anything

if(length(tmp)==0) { Folders[i] <- "" } } ​ Folders <- Folders[Folders!=""] ​ ################################################################################

Liam update (2023-08-04) starts here to edit MD json files (remove erroneous entries) and remove ghost jpeg files

library(jsonlite) ################################################################################ ​ ###############################################################################

Remove entries in json that contain certain file name strings

################################################################################ ​ for(i in 1:length(Folders)) {

Read in the files

tmp.locs <- list.files(path = paste0(renamed_location, "//",Folders[i]), full.names = T, include.dirs = T)

Reads in the JSON, with important assumption that it is always the final file in the list!

tmp.json <-read_json(tmp.locs[length(tmp.locs)], simplfyVector = TRUE )

filtered_images <- list()

Iterate through each image entry in json and filter based on removing NA__NA from the filename string

for (i in seq_along(tmp.json$images)) { if (!grepl("NA__NA", tmp.json$images[[i]]$file)) { filtered_images <- append(filtered_images, list(tmp.json$images[[i]])) } }

Update the parsed data with the filtered images

tmp.json$images <- filtered_images

tmp.json <- toJSON(tmp.json, pretty = TRUE, auto_unbox = TRUE)

writeLines(tmp.json,tmp.locs[length(tmp.locs)])

} ​ ###############################################################################

ghost jpeg delete

################################################################################ ​ for(i in 1:length(Folders)) {

Read in the files

tmp.locs <- list.files(path = paste0(renamed_location, "//",Folders[i]), full.names = T, include.dirs = T)

If you have any ghost jpeg files with "NA__NA" string - remove and delete

for.del <- tmp.locs[grepl("NA__NA", tmp.locs)] if(length(for.del)>0) {

remove it from your list

tmp.locs <- tmp.locs[tmp.locs!=for.del]
# Delete it
file.remove(for.del)

} } ​ ​ ################################################################################

Liam update end

###############################################################################`