Product images missing - after V2.2.0 deployment (?)

tschumilas commented 5 years ago

Description

Most (not all) product images are missing on OFN-CAN.

Expected Behavior

Uploaded product images should be there.

Actual Behaviour

No images.

Steps to Reproduce

1.Log onto OFN Canada, see missing photos 2. 3. 4.

Animated Gif/Screenshot

If I go to the product edit screen to check the photo, I see the thumbnail file - but the photo is not showing in the backend or the frontend.

AwesomeScreenshot-Images-OFN-Administration-2019-07-12-14-07-34

Context

Producers are all pretty ticked off at me today. Lots of work to re-load photos when they need to be out in the fields. This on top of the other cart edit, orders edit, and '0 quantity' bugs is likely going to result in us loosing at least one hub. No one is happy here.

Severity

Your Environment

Version used:
Browser name and version:
Operating System and version (desktop or mobile):
OFN Platform instance where you discovered the bug, and which version of the software they are using.

Possible Fix

luisramos0 commented 5 years ago

They dont need to re-upload the images. The images are in the server. We just need to fix the problem. I dont know what is causing this but I am sure the images are in the server.

For this product, master variant id 9280

openfoodnetwork=> select * from spree_assets where viewable_id = 9280;
  id  | viewable_id | attachment_width | attachment_height | attachment_file_size | position | viewable_type  | attachment_content_type | attachment_file_name |     type     |   attachment_updated_at    | alt
------+-------------+------------------+-------------------+----------------------+----------+----------------+-------------------------+----------------------+--------------+----------------------------+-----
 2785 |        9280 |              225 |               225 |                11186 |        1 | Spree::Variant | image/jpeg              | cherry_brandy.jpg    | Spree::Image | 2019-07-12 18:38:31.801623 |
(1 row)

openfoodnetwork=> \q
openfoodnetwork@ca-prod:~/apps/openfoodnetwork/current$ ls -lart public/spree/products/2785/original/
total 20
-rw-r--r-- 1 openfoodnetwork openfoodnetwork 11186 Jul 12 18:38 cherry_brandy.jpg

Is your example product still not showing the image? Could be a cache problem somewhere...

tschumilas commented 5 years ago

My example product above is now showing an image, but that could be because I uploaded a new image file just to make sure that process was working. One of the things I've noticed is that images for products entered a long time ago are showing. it seems to be more recently uploaded images that are not showing. Not sure if that means anything - or maybe I'm wrong. it just seems like that with my own products anyway.

kirstenalarsen commented 5 years ago

is anyone looking at this? like now? It's really very bad form to lose images on a deploy?!

mkllnk commented 5 years ago

I'll look at it now.

mkllnk commented 5 years ago

I tried to find the missing images in the backup files but they are not there. I may have found a pattern though. ~~All existing images are jpg files and all missing images are png files. But what could have deleted all png files?~~

mkllnk commented 5 years ago

Quick impact analysis. There are 2644 image entries in the database. 1353 really exist and 1291 are missing. Code:

Spree::Image.count
existing = []; missing = []; Spree::Image.find_each { |i| i.attachment.exists? ? existing << i : missing << i }

mkllnk commented 5 years ago

It was probably random that all open shops had missing pngs but existing jpgs. The numbers don't back that up:

existing => {"image/jpeg"=>1283, "image/png"=>70}
missing => => {"image/png"=>104, "image/jpeg"=>1174, "application/octet-stream"=>2, "image/gif"=>11}

More than half of the pngs got deleted and less than half of the jpgs got deleted. All binary and gif attachments got deleted. I'm not sure if that's significant or random.

mkllnk commented 5 years ago

Let's look at date ranges:

Existing: Fri, 19 Jan 2018 - Fri, 12 Jul 2019 Missing: Thu, 03 Dec 2015 - Thu, 11 Jul 2019

Interesting that all images older than Jan 2018 are missing. In fact, there are 1059 entries before 19 Jan 2018 which is 82% of all missing images.

mkllnk commented 5 years ago

And let's look at file sizes. I'm just trying to find a pattern to have a guess what deleted these images.

Existing: 2040 - 9399958 Missing: 2101 - 7713076

No pattern here.

kristinalim commented 5 years ago

Investigating now too.

I'm not seeing anything suspicious in the shell history of both ofn-admin and openfoodnetwork users.

mkllnk commented 5 years ago

Other servers:

Germany: 1 of 3 images is missing. Belgium: 0 of 128 images missing. Using AWS S3 for image hosting. Katuma: 87 of 572 images missing.

I had a quick look at the release notes but I can't find anything related.

mkllnk commented 5 years ago

I'll stop investigating here for now. My summary:

1291 images on the Canadian server are lost. 87 on Katuma. There are no backups.
Instances hosting images on S3 are safe.
The deletion probably happened on file level. Within Rails, the S3 images would have been deleted as well.
It could be a deployment issue because Germany and Katuma are missing images, too.
It's probably not a config issue on the deployer's machine because Canada was deployed by me and Katuma by Luis.
It could somehow relate to the age of the images but it's fuzzy. Katuma lost only newer images after 7 May 2019. Maybe random.

Actions:

Ask Scaleway and Digital Ocean if they can recover data.
Configure backups for Canada and Katuma.
Clone a server and simulate the deploy trying to reproduce the issue.

kristinalim commented 5 years ago

I'm wondering if this is a file system write issue during deployment. The current release's directory for product images has paths for only 1374 products, while for the previous release it is 1567.

$ ls current/public/spree/products/ | wc -l
1374
$ ls releases-old/2019-07-12-045137/public/spree/products/ | wc -l
1567

sauloperez commented 5 years ago

Ok, as suspected the files were not gone but in a different folder, the symlinking of folders involved in the deploy it's always been a bit confusing but now it's broken. Investigating with @Matt-Yorkley we found out that current/public/spree/ it does not symlink to shared/spree and therefore images are not kept between deploys. Fortunately, they do get copied in rollback/ in every deploy.

Next week we'll have to fix ofn-install's symlinking but also, investigate what may have happened to older images that have ended up in releases_old/ or if any images where totally lost between deploys.

@Matt-Yorkley anything else you might want to add?

sauloperez commented 5 years ago

I run cp -R rollback/public/spree/products/* current/public/spree/products/. This brought them to life.

sauloperez commented 5 years ago

@kirstenalarsen @RachL @lin-d-hop and the others in this thread shall we discuss about on-call processes, even if lightweight? at least, we might need a postmortem/retro.

Matt-Yorkley commented 5 years ago

There were missing enterprise images as well, rollback/public/images/*. They are now restored. We should check Katuma.

Matt-Yorkley commented 5 years ago

I'm working on fixing the actual issue now, and adding new tests in ofn-install.

tschumilas commented 5 years ago

Images are back it seems. I see a few missing - but maybe they never had images. Am passing on big hugs and thanks from all the Canadian farmers on OFN - who can now go back to their fields instead of spending the morning on their computers!!! I presume I can close - but will just wait a tad to hear anything further on this.

mkllnk commented 5 years ago

Thank you so much for finding the images! I don't quite understand why I couldn't find them. I looked in the rollback and the old releases folders and couldn't find them. Is there something wrong with this command?

find / -type f -iname 'example-image-name.jpg'

I did that for existing images and I could find them. But the missing images were not found.

Matt-Yorkley commented 5 years ago

We found the images and copied them across as a temporary fix. I've now fixed it properly in ofn-install and added tests here: https://github.com/openfoodfoundation/ofn-install/pull/471

sauloperez commented 5 years ago

Update regarding Katuma. As @mkllnk reported, we also lost some pictures but it turns out our server does not have a rollbacks/ folder but it does have releases-old/. I tried with a particular picture I saw it was missing from /admin/products and I found it in the previous release, 2019-07-12-094854. I went on with another one but this time I found it in the one before the last (using find releases-old/ -name 512).

There were lost images in all releases so I ended up running cp -r releases-old/*/public/spree/products/* current/public/spree/products/ and it did the trick except for at least 4, that I know of :sob: . I'm afraid the fell of the window 5 previous releases we keep :see_no_evil: .

We will also have to check Canada's previous releases as there might be some pictures we have not recovered yet. My last command might do the trick.

I wonder however @Matt-Yorkley why Canada had a rollback/ dir and Katuma doesn't? is it because someone executed a rollback some time ago in that server?

mkllnk commented 5 years ago

I now realised why I thought all missing images are lost. I searched only for a few and was unlucky. Thanks to the great @Matt-Yorkley and @kristinalim some images could be restored. But there are still 1073 database entries without corresponding image file on disk. So we restored 1291 - 1073 = 218 images. Most of the lost images could belong to old products and may not be that important.

I just wrote all missing image file names to a file and search for them in the openfoodnetwork folder. Nothing came up. So I guess you already copied images from all older releases, right, @Matt-Yorkley?

existing = []; missing = []; Spree::Image.find_each { |i| i.attachment.exists? ? existing << i : missing << i }
File.write('images', missing.map(&:attachment_file_name).join("\n"))

for i in "`cat images`"; do find ../ -type f -name "$i"; done

So I guess those images are really lost?

Matt-Yorkley commented 5 years ago

I ran cp -R -n releases-old/*/public/spree/products/* current/public/spree/products/ on Canada and it looks like it copied ~40 more. I think that's all we're getting.

sauloperez commented 5 years ago

I'm investigating a little bit more on Katuma before provisioning https://github.com/openfoodfoundation/ofn-install/pull/471 in it, and something to be aware of is that images could have been updated in later releases :warning: so I'd say the latest release should prevail.

I got this when running:

diff -qr current/public/images/enterprises/logos/ releases-old/2019-06-05-074755/public/images/enterprises/logos/

Los archivos current/public/images/enterprises/logos/141/medium/Logo.png y releases-old/2019-06-05-074755/public/images/enterprises/logos/141/medium/Logo.png son distintos
Los archivos current/public/images/enterprises/logos/141/small/Logo.png y releases-old/2019-06-05-074755/public/images/enterprises/logos/141/small/Logo.png son distintos
Los archivos current/public/images/enterprises/logos/141/thumb/Logo.png y releases-old/2019-06-05-074755/public/images/enterprises/logos/141/thumb/Logo.png son distintos
Sólo en current/public/images/enterprises/logos/: 180
Sólo en current/public/images/enterprises/logos/: 181
Sólo en current/public/images/enterprises/logos/: 185
Sólo en current/public/images/enterprises/logos/: 187
Sólo en current/public/images/enterprises/logos/: 188
Sólo en current/public/images/enterprises/logos/: 197
Sólo en current/public/images/enterprises/logos/: 199
Sólo en current/public/images/enterprises/logos/: 202
Sólo en current/public/images/enterprises/logos/: 203

I also learnt about diff -qr.

sauloperez commented 5 years ago

Closing as this has been solved in https://github.com/openfoodfoundation/ofn-install/pull/471

openfoodfoundation / openfoodnetwork