Loot our local cache of images and extract all of the meaningful information we can about them

[keithamoss]

Loot our local cache of images and extract all of the meaningful information we can about them With the aim of being nice to SRO so we don't have to loot ~25GB of image data from them.

Dependencies: None Next: #4, #11 Associated: #8

Process

• Recurseively find all files on our harddrive
• Ignore special files (e.g. .DS_Store)
• Extract what metadata we can (filename, filesize, filetype, pixel per inch, ???)
• Dump it all to a JSON blob on disk with filename as the key (for easier searching)

  Potential Challenges

• That they're not all JPEG files (Can we solve this in OSX with a kind-not:jpeg filter?)
• That some of them are huge (Can we solve this in OSX with sorting on the aforementioned filter?)

[keithamoss]

Data structure:

CREATE TABLE sro_images
(
  id serial NOT NULL,
  filename character varying,
  format character varying,
  mode character varying,
  filesize integer,
  width smallint,
  height smallint,
  info json,
  exif json,
  md5hash character varying,
  photohash character varying,
  cons_folder character varying,
  CONSTRAINT pkey PRIMARY KEY (id)
)
WITH (
  OIDS=FALSE
);
ALTER TABLE sro_images
  OWNER TO keith;

[keithamoss]

Challenges

• Hashing takes ages to run - probably due to the occasional huge file amongst the maps.

TODO

• [x] Determine if we have collected information on all maps on the drive. Did we miss any non .jpg files?
  • Yes - ~20 odd TIF files. They've been added now.
• [x] Sum the filesize to get an accurate reading on how big these are
  • 24.7GB!
• [x] Collect some stats on PPI and average dimensions
  • 6,579 images
  • Average filesize = 3.75mb
  • Average width,height = 5457,4501
  • DPI: 99% 150 DPI, a handful are 300 DPI

[keithamoss]