doesn't scan directory when loading images

[lindensmith]

I've been having some problems similar to other users with loading images. I finally found and followed your recipe in Issue #6 and got it to load products and images from 'SpreeProductsWithImages.csv'. It doesn't seem to work right when I try to do the images separately.

My file arrangement is:

spreestore/ (rails root) spreestore/db/datashift/SpreeProductsWithImages.csv spreestore/fixtures/images/{the image files}

SpreeProductsWithImages.csv has paths that are: 'fixtures/images/filename.jpeg'

So I sorted out that the file path is relative to where thor is running from, rather than from the directory the csv is in.

When I try to do it by loading products first using 'SpreeProducts.csv' (in the same directory as 'SpreeProductsWithImages.csv') I can load the products just fine, but then when I try to load images with:

bundle exec thor datashift:spree:images -i fixtures/images -s

I get this set of messages and no images in the store: Using SKU for lookup CONFIG: {"split_file_nameon"=>"", "input"=>"fixtures/images", "sku"=>"sku", "load"=>false} OPTIONS {"split_file_nameon"=>"", "input"=>"fixtures/images", "sku"=>"sku"} Will scan image names splitting on delimiter : _ WARNING : 1 of 1 images could not be attached to a Product Copying all images with MISSING Records to ./MissingRecords >> Copy fixtures/images/DEMO_002_Powerstation.jpg to MissingRecords folder

I'm running Rails 3.2.6, Spree 1.1.2, and datashift 0.80 from git. I've tweaking the datashift docs a bit as I figure out how it works-- I'd like to extend it to use it for loading products and digital files for spree_digital.

[autotelik]

Hi there   it looks from the log  like you have a single image file fixtures/images/DEMO_002Powerstation.jpg   The lookup is based on the SKU and it progresivly tries combinations, splitting the name on ''  so  seems like there is no product with a SKU of DEMO or DEMO_002 or DEMO_002_Powerstation

The lookup is quite configurable, see bundle exec thor help datashift:spree:images for info on options   I guess your using the example fixtures rather than your own product & image set ?   Check that there's a Product loaded from 'SpreeProducts.csv' with a SKU of DEMO_002   Cheers tom  

---

From: lindensmith reply@reply.github.com To: tom statter github@autotelik.co.uk Sent: Wednesday, 1 August 2012, 9:07 Subject: [datashift] doesn't scan directory when loading images (#9)

I've been having some problems similar to other users with loading images.  I finally found and followed your recipe in Issue #6 and got it to load products and images from 'SpreeProductsWithImages.csv'.  It doesn't seem to work right when I try to do the images separately.

My file arrangement is:

spreestore/ (rails root) spreestore/db/datashift/SpreeProductsWithImages.csv spreestore/fixtures/images/{the image files}

SpreeProductsWithImages.csv has paths that are: 'fixtures/images/filename.jpeg'

So I sorted out that the file path is relative to where thor is running from, rather than from the directory the csv is in.

When I try to do it by loading products first using 'SpreeProducts.csv' (in the same directory as 'SpreeProductsWithImages.csv') I can load the products just fine, but then when I try to load images with:

bundle exec thor datashift:spree:images -i fixtures/images -s

I get this set of messages and no images in the store: Using SKU for lookup CONFIG: {"split_file_nameon"=>"", "input"=>"fixtures/images", "sku"=>"sku", "load"=>false} OPTIONS {"split_file_nameon"=>"", "input"=>"fixtures/images", "sku"=>"sku"} Will scan image names splitting on delimiter : _ WARNING : 1 of 1 images could not be attached to a Product Copying all images with MISSING Records to ./MissingRecords >> Copy fixtures/images/DEMO_002_Powerstation.jpg to MissingRecords folder

I'm running Rails 3.2.6, Spree 1.1.2, and datashift 0.80 from git.  I've tweaking the datashift docs a bit as I figure out how it works-- I'd like to extend it to use it for loading products and digital files for spree_digital.

---

Reply to this email directly or view it on GitHub: https://github.com/autotelik/datashift/issues/9

[lindensmith]

Thanks for the reply.

I'm just using the example files and fixtures (and have cleaned out all the other products from the database).

So I have Products with SKU's of DEMO_001, DEMO_002, and DEMO_003, and images in the fixtures directory that start with DEMO_001, DEMO_002, and DEMO_003, and DEMO_004. When I load them using SpreeProductsWithImages.csv, all three load. When I load them from SPREE_products, it gives me the message about DEMO_002, and ignores the existence of the other images.

[autotelik]

No probs, thanks for feedback, helps improve things.

Strange since it only warns about the one image, says 1 of 1 failed to find a product.

It writes log messages to standard rails log so that might help us track down issue. Use -v or --verbose for more logging.

Ill try to mimic your setup tonight, im using this heavily on projects at moment and seems ok

Tom

Sent from Yahoo! Mail on Android

[lindensmith]

I sorted it out-- it was two things: 1) the current rev detects .jpg, .gif, and .png, and 3 of my 4 had ".jpeg" with the "e" so they weren't getting seen. 2) it splits on "_" and doesn't do recombination, so if the SKU is "DEMO_001" it looks for "DEMO" and "002" but not "DEMO002". The solution is to either use the prefix or not have "" in the SKU.

I added jpeg (with an e) to the list of files it looks for, and made a note in the docs about the "_". I'll send you a pull request if you want to incorporate them.

[autotelik]

Hi Linden   glad you got it sorted.  Thanks for feedback, definitly up for a pull request.   I realised recently  that glob is  far too restrictive, I will try to make it match anything  that imagmagic will process   Cheers tom

---

From: lindensmith reply@reply.github.com To: tom statter github@autotelik.co.uk Sent: Friday, 3 August 2012, 18:36 Subject: Re: [datashift] doesn't scan directory when loading images (#9)

I sorted it out-- it was two things: 1) the current rev detects .jpg, .gif, and .png, and 3 of my 4 had ".jpeg" with the "e" so they weren't getting seen. 2) it splits on "_" and doesn't do recombination, so if the SKU is "DEMO_001" it looks for "DEMO" and "002" but not "DEMO002".  The solution is to either use the prefix or not have "" in the SKU.

I added jpeg (with an e) to the list of files it looks for, and made a note in the docs about the "_".  I'll send you a pull request if you want to incorporate them.

---

Reply to this email directly or view it on GitHub: https://github.com/autotelik/datashift/issues/9#issuecomment-7489182

[lindensmith]

I'm not sure that glob is that much of a problem as long as it's documented what to change to change what loads. I copied all the image-loading code and walked through it line by line to make a loader for general digital documents (specifically for my fork of spree_digitals) and it turned out that very little code had to be changed. Aside from the model name, the set of file endings was one of the changes, but that makes it convenient to limit epub, mobi, and pdf if I want. Or mp3, flac, and ogg for someone who wants digital downloads.

[autotelik]

Hi

This sounds really interesting as I am currently working on a site for my dad whose an artist which was big driver for datashift image loading but I need to add the downloadable link aspect

Definitely be interested in merging your additions into datashift and adding you as a contributor

---

From: lindensmith notifications@github.com To: autotelik/datashift datashift@noreply.github.com Cc: tom statter github@autotelik.co.uk Sent: Monday, 6 August 2012, 22:40 Subject: Re: [datashift] doesn't scan directory when loading images (#9)

I'm not sure that glob is that much of a problem as long as it's documented what to change to change what loads. I copied all the image-loading code and walked through it line by line to make a loader for general digital documents (specifically for my fork of spree_digitals) and it turned out that very little code had to be changed. Aside from the model name, the set of file endings was one of the changes, but that makes it convenient to limit epub, mobi, and pdf if I want. Or mp3, flac, and ogg for someone who wants digital downloads. — Reply to this email directly or view it on GitHub.

[lindensmith]

spree_digital works pretty well. the main fork sends an email to a link that's available for one download up to 24 hours after purchase. I'm doing e-books and want to leave the user a bookshelf, so I send the link but did a mod so the user can go back to look at their orders and re-download things. It has a slightly odd feature (useful in some ways, a pain in others) where the user can only see the versions that were available at the time of purchase, so if they buy it when there's just .epub, and then .pdf and .mobi become available, they won't see those. It's good for some kinds of stores, bad for others. I also added a feature to have free samples (maybe less important for art, where the product image is likely the sample, but readers need to be able to read a sample) that are linked to each product.

The way I have the digital loading working now, it's extremely non-DRY (you can see if you look in my branch called spree_digitals)-- basically a copy of the image loader with very small tweaks. I'd like to clean it up so it's just the same code with a switch instead of duplicate code before I submit the pull request. I'll send a pull for the small changes (.jpeg and to the readme) right away.

[autotelik]

FYI I've also checked some changes in including changes to the glob processing

I've been slowly realising that image loading badly named and should become something  like attachment loaderand your right, have some way of determining what type of attachment is being dealt with and react appropriately

---

From: lindensmith notifications@github.com To: autotelik/datashift datashift@noreply.github.com Cc: tom statter github@autotelik.co.uk Sent: Tuesday, 7 August 2012, 15:40 Subject: Re: [datashift] doesn't scan directory when loading images (#9)

spree_digital works pretty well. the main fork sends an email to a link that's available for one download up to 24 hours after purchase. I'm doing e-books and want to leave the user a bookshelf, so I send the link but did a mod so the user can go back to look at their orders and re-download things. It has a slightly odd feature (useful in some ways, a pain in others) where the user can only see the versions that were available at the time of purchase, so if they buy it when there's just .epub, and then .pdf and .mobi become available, they won't see those. It's good for some kinds of stores, bad for others. I also added a feature to have free samples (maybe less important for art, where the product image is likely the sample, but readers need to be able to read a sample) that are linked to each product. The way I have the digital loading working now, it's extremely non-DRY (you can see if you look in my branch called spree_digitals)-- basically a copy of the image loader with very small tweaks. I'd like to clean it up so it's just the same code with a switch instead of duplicate code before I submit the pull request. I'll send a pull for the small changes (.jpeg and to the readme) right away. — Reply to this email directly or view it on GitHub.

[lindensmith]

I just saw that-- I think you probably did everything I was going to push.

The attachment changes are really simple-- it's not obvious from my commits in the digital loader path. It's a command line option to pick the model, and if the model isn't image then it attaches to the variant instead of the master variant. I also added code to the cleanup to clear out the tables spree_digitals and spree_digital samples. If you don't beat me to it, I'll see if I can get it done tonight.

[lindensmith]

I finally got around to cleaning up my digital loader-- I just pushed it to my spree_digital branch. I tore out all the code that I had duplicated and put in a '-m' option to specify the model (defaulting to Image) and put in appropriate switches to load images differently than digital models (e.g. spree_digital doesn't have a viewable parameter). It needs another switch to reconcile the file types (use your ImageMagick glob for images, and use other endings for digitals (epub, mobi, pdf, etc))

Also still need to verify that I didn't break the loading of products with images.

[lindensmith]

latest update is verified to load products with images properly.