search

nulib / images

Hydra based image application
https://images.northwestern.edu/
3 stars 1 forks source link

Remove extraneous images from the dimages dir #389

Closed carrickr closed 7 years ago

carrickr commented 7 years ago

Any jp2 that doesn't exist in solr as a multires image does not need to be converted to .tiff. Find a location to move these rather than convert them.

jenyoung commented 7 years ago

please don't forget the missing tiffs that need to be found https://github.com/nulib/repodev_planning_and_docs/issues/18

@Nic4Images

carrickr commented 7 years ago

@jenyoung For display purposes we'll just use the jp2 display copies to make our display tiffs. This issue is dealing with improving performance.

carrickr commented 7 years ago

We've moved all .jp2s that don't have a solr record to orphans.

@davidschober We need a final call on what to do with the .jp2s

davidschober commented 7 years ago

What's our confidence level that these were deleted from images (e.g. there's no associated metadata)?

bmquinn commented 7 years ago

Most of the orphans are likely from either deleted image records, batch testing or batch retries in sidekiq ( which used to create many orphaned derivatives during retry cycles but is now fixed)

davidschober commented 7 years ago

Thanks. I'll dig into this a bit with the stakeholders. let's get a final size count on data when we're done. It sounds like our orphans were due to the system.

Nic4Images commented 7 years ago

If the metadata is not there for these then it sounds okay to get rid of them.

jenyoung commented 7 years ago

i agree with Nicole. No metadata, no problem deleting

davidschober commented 7 years ago

k. Let's get a count when we're done.

carrickr commented 7 years ago

Those without a record are:

oprhans.txt

Counnt: 38759

davidschober commented 7 years ago

Cool. Let’s hold these for now.

On May 1, 2017, at 5:29 PM, Carrick Rogers notifications@github.com wrote:

Those without a record are:

oprhans.txt https://github.com/nulib/images/files/968934/oprhans.txt Counnt: 38759

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nulib/images/issues/389#issuecomment-298449858, or mute the thread https://github.com/notifications/unsubscribe-auth/AElKntOlmFyNpMQhciyPyIYGcfdqFF39ks5r1lzQgaJpZM4NLkLn.