Downloading/Uploading levels from API

[jvyden]

Will be extremely helpful for archival efforts

Users are opted in by default, but if they choose they can disable downloads for their levels

[Arcadius2006]

downloading should only be enabled by setting a level to be copyable

levels owners will be able to download any of their levels, and admins can download any level

[Arcadius2006]

and moderators, can also download any level too, ig

[GraysonMakesGames]

downloading should only be enabled by setting a level to be copyable

levels owners will be able to download any of their levels, and admins can download any level

if this is from the API and not the game itself, how can we check for that? sorry for the stupid question

[speedyman08]

levels owners will be able to download any of their levels, and admins can download any level

For this to happen there should be some sort of authentication system implemented where you could get a temporary auth key that you could send with the API request, to check if your account is the owner of the level.

[speedyman08]

Every level should also have a unique Level ID, but i'm not sure if Refresh has this already

[jvyden]

levels owners will be able to download any of their levels, and admins can download any level

For this to happen there should be some sort of authentication system implemented where you could get a temporary auth key that you could send with the API request, to check if your account is the owner of the level.

This is already done, and we can use that system for this if we want to.

Is the general consensus that it's not okay for a level to be downloaded by default? I was hoping not, as it would impede a bit with archival efforts. The reason I ask is because if a Refresh instance goes down, unless the owner of the server releases their resource folder then that server's levels are gonna be gone forever.

I get the whole privacy bit, but it's honestly not that difficult for someone to dump your level if they really wanted to (especially now that Refresh offers API endpoints for downloading resource hashes)

Not a final plan yet of course, so I'm willing to discuss if you have other opinions (particularly Arek)

[speedyman08]

I don't really have a problem with levels being able to be downloaded by anyone, but some bad actors could download levels and just reupload them on the server which would be annoying

[jvyden]

We can definitely write some checks to look for duplicate levels (e.g. only one level with a given RootLevel can be published) but past that I think it's a moderation issue

[Arcadius2006]

i think administrators being able to download every level would be good enough for archival, as one could just keep an archive of all the levels published

you could even add an option to download levels on mass for them to make the job easiee

[jvyden]

It might be good enough, yes, but on the other hand why limit that functionality to administrators? I personally see no reason to restrict the ability to archive a server's contents if you want to. In my opinion it should be always be able to be done externally.

For example, the official servers aren't able to be archived right now unless Sumo releases something, which is unlikely. Neither will LBPU's Beacon, probably.

Those servers have a lot of content that could be lost if they go down. And they definitely will at some point, and so will Refresh. It's the administrators choice to release the full dump. But I think it should also be in the hands of anyone else to keep an archive of some sort on-hand just in-case.

Also, once again, people can still log in and forge requests as the game client to download levels en masse. This is about making it easier. :p

[jvyden]

you could even add an option to download levels on mass for them to make the job easiee

This is already done technically, you can just make a copy of your dataStore folder & Realm and you'd have all the server's level contents with metadata. That said, there could be a tool to let you share it easier as the Realm contains sensitive data.

[Arcadius2006]

i also reccomend making the downloaded levels ready to use level backups, if possible

[turecross321]

I think that all archived levels should be downloadable to everyone, at the very least.

[CrankyLandlord]

Honestly feel like there has to be as high a barrier for entry for copying from the archive as possible--I like the idea of an admin only file with access to all the levels instead. I know the reasoning for easy access is primarily for preservation but to me having directly copyable in game or on the website is kind of just asking for swarms of non-transformative content.

Can see once users figure out 'oh I just need to slightly alter this level and then I'll be able to republish' (or however the automated system disallows reups) then it become this task of moderation having to manually play swaths of levels that are simillar to the originals with different hashes, to confirm.

I'm not saying the average user who would do this is malicious but more just they won't read rules or won't understand the ettiquette of 'it's not cool to copy levels'."I want to publish this really cool level but I can't, I'm going to fiddle with it in moon, oh ok now I can, weird"

[Beyley]

I'm not saying the average user who would do this is malicious but more just they won't read rules or won't understand the ettiquette of 'it's not cool to copy levels'."I want to publish this really cool level but I can't, I'm going to fiddle with it in moon, oh ok now I can, weird"

This is an issue solvable through saying "dont do this" to them and deleting the level, if they are a repeat offender, punishing them with warnings/restrictions

I feel like we shouldn't compromise on an archival feature because it can be misused to spam reuploads, which is going to happen until the end of time anyway. Given we have still lost all the levels from after the dry archive but before the server shutdown, we shouldn't take a chance, and should make it as easy as possible to archive levels

i also reccomend making the downloaded levels ready to use level backups, if possible

I feel like we shouldn't provide these downloads as game level backups, but rather a simple custom format, as making a proper level backup is a lot of complexity for little to no benefit. If we provide ways to upload a level using said custom level package format (like a zip file containing a json blob [can use the same serialization as the API] and all the assets), then its easy for people in the future who are wanting to use these individual level archives to put them onto their own server, there could even be a mass import tool in Refresh

i think the simplest way is GET /api/v3/levels/{id}/download -> ZIP file containing the following structure:

103.json # level metadata in JSON form, so name, description, icon hash, root hash, the name is the level ID
assets/{SHA1HASH}
assets/{SHA1HASH}
assets/{SHA1HASH}
assets/{SHA1HASH}
assets/{SHA1HASH}

The reason the level ID is given for the JSON file is so that these level packages could be directly merged with each-other to create a level package containing all the levels of a specific user (so you can easily export all your own levels into a single big zip)

So the proposed flow for import is

• API Client decompresses level package
• API Client uploads all the assets to the server
• API Client sends the processed JSON blob to /api/v3/levels/upload

Although another path we can take is that the creation of these ZIP files can be the responsibility of the API Clients and not the server, creating a ZIP file isn't exactly the cheapest thing you can do, and all of the functionality to implement this externally is readily exposed already by the server, so maybe there should be an issue opened in refresh-web for downloading level packages (where JavaScript code is run on the client to construct the package using existing level and asset APIs), and then this issue can be dedicated to just a theoretical /api/v3/level/upload endpoint to allow uploading levels through the v3 API.

[Arcadius2006]

oh? this is back up again? well i agree that all archive levels should probably be downloadable, as it's already possible to do that with various tools

still think actual new levels should only be downloadable if they're set to copyable though, since it's basically the same action as copying

[CrankyLandlord]

If we give everything away, even if we include the dupe hash blocker we're exponentially increasing the number of moderation we have to do, and Refresh really does not have the staff to be taking down seas of copied levels--we don't do it enough as is. I know that eventually that will be a role on the website, but really even if we have more mods, we're just adding work for ourselves, especially by making copying levels that much easier. The lack of duplicate hashes would also mean that, each instance of a suspected dupe, a mod needs to manually play the level to check for similarities--there are plenty of instances where levels are, for better or so so often worse, transformative, so we can't just nuke it from orbit if we care about fairness.

Aside from that, I'd say making everything readily available really has no benefit that 99% of people will properly use. 1) 'copy levels to moon for a personal copy if servers go offline'--already possible via other tools 2) 'modifying a version of the level'--most people who do this and publish really don't add anything worth adding. If you check beacon there are so so many copies of story levels or sometimes community levels that are just unplayable destroyed versions of other levels. 3) 'learning from logic in create mode' actually a genuinely good use but there are only a handful of people on here who actually try to create. 4) 'publishing your old levels to your earth' could be done by admins, inspires people to learn that 'if you go into create and modify lvl a little bit you can republish' perpetuating potential issues.

I think this, inability to delete comments, infinite level slots ruining newest (ik we're adding data cap but still), we're thinking of these things as if our instance is super small and off the map. If we hand people we don't know the tools to break things often they do, and there really aren't enough benefits to give them those tools.

[Beyley]

still think actual new levels should only be downloadable if they're set to copyable though, since it's basically the same action as copying

This has been mentioned by me and jvyden in the past in this thread, but "copyable" is a lie in LBP. Nothing prevents someone from just downloading the assets and putting it on their profile (see the probably hundreds of levels from the dry archive reuploaded to LittleBigRefresh). You dont even need to pretend to be a game client, this is already possible through the public Refresh API by design. This is how tools like lbp_archive_dl work, they use Refresh as a source to download level assets from the dry archive (since we re-host the whole thing).

If we give everything away, even if we include the dupe hash blocker we're exponentially increasing the number of moderation we have to do, and Refresh really does not have the staff to be taking down seas of copied levels

If you read the second half of my comment where I respond to Arcadius, you see I explicitly propose not using normal level backups here. This achieves 2 goals, it simplifies things greatly, but it also prevents people from importing these level packages into their game, so they will be unable to just "accidentally" modify it slightly to re-upload. Duplicate hash checks are still going to take place for uploads done through the API. Someone would have to intentionally work to bypass these checks using a tool like toolkit to re-serialize the assets so the hash changes, in which case that is a moderation issue and the user should be published.