search

scratchfoundation / scratch-flash

Open source version of the Scratch 2.0 project editor. This is the basis for the online and offline versions of Scratch found on the website.
https://scratch.mit.edu
GNU General Public License v2.0
1.33k stars 512 forks source link

Disappearing assets – data collection #403

Closed Paddle2See closed 9 years ago

Paddle2See commented 10 years ago

This issue is to be used to collect information on reports of missing assets (costumes, sprites, sounds, scripts) in an effort to see a pattern and hopefully fix the cause or causes.

Here is one I came across today that reports a large loss of data

http://scratch.mit.edu/discuss/post/385608/

interestingly, the person also reports that the assets appear to be missing from remixes as well (see two posts below).

There is another report at the top of this topic as well

http://scratch.mit.edu/discuss/post/384158/

nathan commented 10 years ago

Almost certainly caused by #138.

nathan commented 10 years ago

Broken PNGs:

nathan commented 10 years ago

It's possible we're saving the base layer data from https://github.com/LLK/scratch-flash/blob/9be7d2e86e5c90a1c1d6460a0049ae307f1d74b0/src/util/ProjectIO.as#L412 without recomputing the MD5 (for some reason).

sclements commented 10 years ago

I agree, that looks like it could be overwriting the content but keeping the same MD5 which could later save the blank SVG using the original MD5. We should probably be using the setSVGData() method instead.

Paddle2See commented 10 years ago

Another incident: This user reports that some of the costumes on her movie-clip projects have suddenly gone blank (white flashes as the project plays). As an example, she gives this project

http://scratch.mit.edu/projects/16465251

After watching the animation, I found that the sprite "Elsa Pressure", Costume labeled "New Picture (7)" appears as blank 0x0 pixels. Presumably, it used to have content. I've asked for more examples.

She says this has happened on other projects recently - but she already fixed those by reloading the images.

Paddle2See commented 10 years ago

Here is a case where several sprites have had costumes blanked out

http://scratch.mit.edu/projects/18304058/

The user reports that they have also disappeared from the remixes too - since we aren't sharing assets between projects (are we???), this perhaps implies a wide-spread issue affecting a class of costumes. Here is one such fairly recent remix

http://scratch.mit.edu/projects/23096056/

that has blank costumes

lightnin commented 10 years ago

@sclements Since this involves losing user's work, I'd like to prioritize #138 for fixing / hotfixing in the next few days. @nathan is in Colorodo this week and probably won't have time to sort it out. Can you or @cwillisf take a look at this today or tomorrow?

lightnin commented 10 years ago

Inaccessible backdrop noted by Natalie in devs-scratchr2 email subject "empty backdrop 0x0" Can anyone tell me how to identify if this is likely related to #138 ?

WF saving project id: 25114791 owner: natalie title: Try Scratch remix project saved! (401 msecs) Failed to load resource: the server responded with a status of 403 (Forbidden) http://cdn.scratch.mit.edu/internalapi/thumbnail/120/90/c3464cbe80c4d3adb225383d9e84b5d4.png Failed to load resource: the server responded with a status of 403 (Forbidden) http://cdn.scratch.mit.edu/internalapi/thumbnail/120/90/2dbde2ae53f2744b87be09581281610f.png SWF saving project id: 25114791 owner: natalie title: Try Scratch remix project saved! (9800 msecs) Failed to load resource: the server responded with a status of 403 (Forbidden) http://cdn.scratch.mit.edu/internalapi/thumbnail/120/90/c3464cbe80c4d3adb225383d9e84b5d4.png

sayamindu commented 10 years ago

The 403s were a result of missing thumbnails for some of the media library stuff. I fixed those.

The brick wall seems to work now, so no idea what's going on there. The missing brick wall and the 403s are probably unrelated.

nathan commented 10 years ago

since we aren't sharing assets between projects (are we???)

We are.

nathan commented 10 years ago

http://scratch.mit.edu/projects/18304058/

Broken PNGs:

The remix has exactly the same list. (script I'm using for this)

sayamindu commented 10 years ago

All the broken PNGs above seem to be blank SVGs.

On Tue, Jul 29, 2014 at 3:27 PM, Nathan Dinsmore notifications@github.com wrote:

http://scratch.mit.edu/projects/18304058/

Broken assets:

— Reply to this email directly or view it on GitHub https://github.com/LLK/scratch-flash/issues/403#issuecomment-50525714.

nathan commented 10 years ago

All the broken PNGs above seem to be blank SVGs.

Yes. Can you try what Shane did?

[Jul-21 12:38] Shane M. Clements:

shanemc@scratch-assets:/data/assetstore/c3/c3e9/c3e9f8a5$ sudo cat c3e9f8a54883b5e236067db2a453d7af.png

<svg width="0" height="0"

  xmlns="http://www.w3.org/2000/svg" version="1.1"

  xmlns:xlink="http://www.w3.org/1999/xlink">

</svg>

… [Jul-21 12:39] Shane M. Clements: 

shanemc@scratch-assets:/data/assetstore/c3/c3e9/c3e9f8a5$ cat /data/assetstore-old/c3/c3e9/c3e9f8a5/c3e9f8a54883b5e236067db2a453d7af.png
�PNG

IHDR@��~� pHYs%%IR$��IDATx���1
�0����
)�{�_ 8�9�x!:&S\�����D��^Fҵ�
����*���*�K��o>������is(��0��bљ�X��U{@�y\Ny�a�!���v��s�v�oI���Fk���:�s����$c�(k-Iz��k��a(�/B(c�R"��I�@ 7w��IEND�B`�

(If that happens for all of them it's a good thing because it means we didn't permanently lose any of the assets)

sayamindu commented 10 years ago

Yep - done. All assets in that list are now recovered.

Here's what's happening:

We started to store assets in a new server, so even assets that existed earlier (i.e. same MD5s) would get stored (the plan is to start merging the two directories from the two servers together next month). Some of these assets came in as blank SVGs, but it did not show up immediately, since the order of file search was to look at the old server first, and then the new server. We reversed that order recently, and hence all the problematic PNGs are showing up.

Has the SWF bug that sends the broken PNGs been found/fixed?

lightnin commented 10 years ago

Getting more reports on my profile. Here's one that seems legit so far: http://screencloud.net/v/5JYi

Nothing in console. http://scratch.mit.edu/projects/25091295/#editor

lightnin commented 10 years ago

@sclements Can you update us on status / questions from @sayamindu in above comment?

Paddle2See commented 10 years ago

Here is a case where Scratch froze during saving - project now appears as blank

http://scratch.mit.edu/projects/25180893/

Forum report

http://scratch.mit.edu/discuss/topic/47599

Would somebody check and see if there are any recoverable assets? Is this something I can do myself, with training?

ntlrsk commented 10 years ago

This mini-fig project by Ericr and its remix by Scratchteam has a missing sound ""boing" in Sprite1 (it shows as .30 length but is blank for me). Do you see it as blank?

http://scratch.mit.edu/projects/10118726/

We do have a copy of the sound in the offline projects if we need to reupload -- but I'm thinking that it would be good if possible to recover the original so that it appears in the other remixes of the project.

(Meanwhile the Brick wall image now works for me, thanks!)

Paddle2See commented 10 years ago

I have another project that will not load any more - and neither will it's remixes.

http://scratch.mit.edu/projects/11210299/

this project doesn't appear to generate any JS errors in the console. The remix that I tried

http://scratch.mit.edu/projects/12069029/

gave this error

Failed to load resource: the server responded with a status of 429 (UNKNOWN STATUS CODE) https://app.getsentry.com/api/19106/store/

Can anybody explain what is going on here?

griffpatch commented 10 years ago

I've been having reports of missing costumes from paper minecraft for many months, but assumed it was down to them failing to load due to memory issues or something... but I realise now that it's most likely the same issue reported here with missing assets.

If I load paper minecraft in my usual browser then most of the time there is no problem, but that's because the correct assets are cached locally? Am I right? If I go incognito browser then the same assets fail to load consistently... and LOTS of them. Just look at last 20 costumes of the tiles sprite!

Now, I have a backup of the sb2 file which I tried re-uploading, which looked like it worked, but on using the incognito browser again it still failed, all lost costumes appear 0x0 pixels.

I can email you the correct .sb2 file if it helps with the assets intact.

Griffpatch (andy)

On 19 August 2014 01:52, Paddle2See notifications@github.com wrote:

I have another project that will not load any more - and neither will it's remixes.

http://scratch.mit.edu/projects/11210299/

this project doesn't appear to generate any JS errors in the console. The remix that I tried

http://scratch.mit.edu/projects/12069029/

gave this error

Failed to load resource: the server responded with a status of 429 (UNKNOWN STATUS CODE) https://app.getsentry.com/api/19106/store/

Can anybody explain what is going on here?

— Reply to this email directly or view it on GitHub https://github.com/LLK/scratch-flash/issues/403#issuecomment-52577155.

ntlrsk commented 10 years ago

Claudia Urrea just reported on a missing sound (I wonder if it's similar to the issue withthe missing sound that I reported from the Minifigs project above).

From: Claudia Urrea calla@media.mit.edu Subject: Bird sound

Hi!

One of the teachers in Ecuador sent an email to me about her project not working. She is using “bird” sound, and for some reason the sound appears “empty” in Scratch editor. Is this an known problem? I checked at random and other sounds seem fine.

Thanks,

Claudia

ntlrsk commented 10 years ago

Note the "bird" sound she is mentioning is in the Scratch default media library. It works in the offline editor but is now missing in the online version.

sayamindu commented 10 years ago

@ntlrsk does it happen with the bird sound in all projects, or in only one specific project?

ntlrsk commented 10 years ago

@sayamindu All projects.

sayamindu commented 10 years ago

@mewtaylor's script just finished, and this should be fixed now.

Paddle2See commented 10 years ago

A report of missing costumes in this project

http://scratch.mit.edu/projects/28667698/

Looking at the project, all costumes of all sprites are 0x0 pixel and blank

Forum report

http://scratch.mit.edu/discuss/topic/56816

Paddle2See commented 10 years ago

Many 0x0 missing costumes. Forum topic:

http://scratch.mit.edu/discuss/topic/65264/

Original project that was damaged:

http://scratch.mit.edu/projects/33604676/#editor

Author has been manually repairing the project above - so it might be difficult to see the state of the damage. Apparently, though, he/she preserved a copy of it in this project

http://scratch.mit.edu/projects/34418836/

Perhaps there is a version of either project that can be restored?

When asked if the author had any theories as to how the project became corrupted, the response was:

"Most likely it happened while I was using this without internet connection. I added a few things then immediately when I had internet connection I saved, possibly the whole project had not been loaded and it saved but without the costumes"

sclements commented 10 years ago

Their story makes a lot of sense. I looked at the code with @cwillisf and I think that my pull request will fix their type of situation. However, we should think about what additional ways to communicate to the user about the state of their project when costumes fail to load. Maybe we should even stop them from using / editing the project and say something like "It appears your internet connection to the Scratch servers is unable to load some of this project. Please check your connection and reload the project."

sayamindu commented 10 years ago

Makes a lot of sense to freeze the project if things don't load. What would be the chances of false positives? Also, if it's not too difficult, will it be possible to have a retry-fetch button on the dialog (I imagine flakey wifi connections that take a while to stabilize, causing assets to half-load, but once they do, they are fairly stable)

Paddle2See commented 9 years ago

Here's an older project with a number of 0x0 costumes

http://scratch.mit.edu/projects/21337925/

The forum topic is here

http://scratch.mit.edu/discuss/topic/67526

The project owner is requesting that the project be returned to an earlier save state where the costumes were not missing.

2jour commented 9 years ago

We are closing this issue. If you notice specific issues with regards to data loss then please log a separate issue. We take data loss issues very seriously and it helps us troubleshoot if they are logged separately.