Publish selected apps

[Schnark]

First of all, thank you for this repository. I, too, made a backup, but only of a very limited number of apps (mainly those I use myself, and some that looked interesting), and was planning to publish them in some way (similar to my own apps on https://schnark.github.io/).

Since you only mention legal obstacles for not publishing the backup you created, I think two types of apps can be published without problems:

• apps with an open source license (though it might be difficult to determine this automatically), and
• apps with "installs_allowed_from": ["*"] in their manifest. This value is the default, so setting it explicitly seems to me like the permission to anyone to distribute unmodified copies.

So I'm wondering whether you are willing to publish some selected apps, or whether I should go ahead and publish those that I backupped.

[mossroy]

Yes, the only obstacle is legal.

You're right for apps under a compatible license (NB : the license must allow redistribution, which is not always implied by "open source"). Unfortunately, the Marketplace did not ask for a license on the Marketplace, so there is no such field in their JSON files. So yes : it's not easy to determine this automatically. One way I thought of would be to look for github URLs (there is sometimes in the homepage or support_url fields), then try to fetch the license through the github API (see https://developer.github.com/v3/licenses/#get-the-contents-of-a-repositorys-license). It should be technically possible, but is not 100% reliable (the github URL might be wrong, the github license detection might be wrong too). And it does not seem to match a lot of apps.

Regarding the "installs_allowed_from" field, I don't think the "*" value can be legally considered as a permission to redistribute. At least because the Mozilla documentation you point to says "This field is intended for apps that end-users can purchase".

The only safe approach (IMHO) would be to use the "support_email" field, send emails to ask permission to redistribute, and publish only the apps for which the permission has been given. If someone takes the time to automate this process, it's certainly the best solution. It's the solution that Mozilla gave me when I discussed with them, and it's the only 100% reliable way to avoid legal issues.

[Schnark]

I just found https://archive.org/details/Firefox_Marketplace_2018_03_Capture. So the Internet Archive already has a full backup of the Marketplace, though you have to download 14GB ZIP even if you just want one app. It might be worth to evaluate whether the Internet Archive would also host the backup in a way, where all the apps can be downloaded separately.

[mossroy]

Very interesting! I downloaded it : it really looks like a zip of the content generated by my script. And it looks like the apps can already be downloaded separately through this page https://archive.org/download/Firefox_Marketplace_2018_03_Capture/Firefox_Marketplace_2018_03_Capture.zip/ (even if it's not very convenient as the web page is very huge).

That's cool, but I don't know how legal this is

[Schnark]

Ah, I hadn't seen that page. So, yes, the apps and all the other content actually is available separately already. And as your script is included into that dump, too, it doesn't just look like it was generated using your script.

As the Internet Archive is a real archive according to US law, it certainly has some privileges in hosting copyrighted contents, though I don't know the details.

[wryun]

I decided to be legally dubious and threw the results of your script on the web in a slightly more accessible form:

https://storage.googleapis.com/fxos-marketplace/index.html

If someone chases me, it will be removed, but I'm hoping that no-one really cares (witness sad death of marketplace anyway).

[madb1lly]

@wryun Wow! Nicely done. FYI, I removed the device filter from mossroy's script and downloaded everything so I've also got desktop apps (for posterity). Let me know if you'd like to add them to your upload.

[mossroy]

@wryun : are you sure you have put the whole content? I don't find an app that I developed, called Evopedia, that was on the marketplace in 2 categories (education and another one).

[wryun]

@mossroy ah, that's frustrating - no, I'm not sure. I just quickly ran your script a couple of weeks ago then did:

https://storage.googleapis.com/fxos-marketplace/generate-html.py

(errr... yeah)

If you know the id, you can check to see if it's there, or you can check listing.json in the root (i.e. if the error was in my downloading or in my script).

I'm not linking to listing.json because it's rather large and I'm paying for data on the bucket transfers, but feel free to slurp it out and check. You can also checking the listing-*.json files if you know which one it should be in.

[mossroy]

After investigating with @wryun , we confirmed that its archive of apps is not complete. It happened when the script was run only once, probably because the paging API of the Marketplace did not work as I expected (see README.md of this repo).

But the one of archive.org is complete.