Integration with stardew.info

[hpeinar]

Hi!

I'm the author of stardew.info/planner @GitHub and I'm here to discuss possibilities to link these two tools.

Without going deeply into technical details yet, I'm just here seeking for your ideas & feelings on this idea and if we should discuss it further.

Few pros I would see from this:

• As both of these tools are widely used in the community, bringing them closer together would definitely benefit the community as a whole.
• As upload.farm is missing the "planning" functionality, integration with the planner would offer this.
• As planner is missing the "render image" functionality, integration with upload.farm would offer this.
• Users would have an easy way to start planning their farms with one click from upload.farm.

Few technical details:

• Planner already supports importing your save game file into the planner. This is something upload.farm could use.
• I would see this as "two separated tools linked" which means they would still both live on different domains and be managed by different people but would be connected via links from one to another.
• They are both open source, managed by the community and don't run any ads, so in these areas they would fit well.

I'm open for all ideas & discussion on this topic.

[Laukei]

Hi Henrik, thanks for getting in touch! Ash and I both think that your proposal would benefit both our tools' functionality and the Stardew Valley community, so it seems worthwhile to pursue this.

(apologies, mis-clicked and Comment & Closed at this point, what follows I added after)

In more specific terms, my intuition for this would be that people who upload their savegame to upload.farm would see a button to 'export to stardew.info planner', which would pass them through to your site and load their farm into the tool. People on stardew.info/planner would also see a 'render farm' button, but I'm less clear on how that would work - maybe pass JSON across to upload.farm, which would render the image, and display it on a stripped-down profile page?

It shouldn't be too much work to find some way of passing uploaded farms into your planner. I think we'd need to have a new type of page for people who use upload.farm to render planner designs, as obviously there won't be any character information or stats to display, but I also don't think that's a significant task.

How does this sound?

[hpeinar]

Sounds something like I thought it would be like.

Though as the planner also supports save game import (it currently does not keep the save file though), we could also have "Take your savegame to upload.farm", which would open up all the information your tool provides.

Talking about rendering the planner to a picture, maybe it would make more sense to use Stardew XML format? That way you wouldn't need to make a special parser for the input (JSON) and I could use the same export for both making the picture and actual game save export functionality (which isn't planned right now...but might come)? Technically it would probably work something like this:

1. User presses "Render farm" in the planner
2. Planners server puts together XML representing current state of the farm.
3. XML gets sent to upload.farm which generates url from where the picture is available (upload.farm link) and provides link to it. (instead of just responding with imgur link, providing upload.farm link would make sure people actually end up on your page)
4. Planner would then either redirect user to this url or just show the link for the user

It would miss all the information of course, but the TerrainFeatures etc would be in "game like" format.

This flow could also work in opposite direction:

1. User presses "Export to planner"
2. Your server sends that save XML over to the planner, planner provides link
3. You show / redirect user to that link.

[Laukei]

It's no problem for upload.farm to take Stardew XML format data if you were going to develop the planner to have export functionality. However, at present, when we render the farm we take the XML and reduce it to JSON like this. I don't know how the planner stores its state internally but I assumed it was probably something more easily converted into something like that than a full XML output. I could develop an API that would allow your planner to pass either a barebones JSON or a full XML file as input, so you can switch between them as you progress?

[hpeinar]

Planner currently stores the farms as following: https://stardew.info/api/247f5592-271f-452e-9df1-d3ba3eed5b6c (this is the raw JSON coming from the db).

Generating JSON like you linked, shouldn't be a problem. To be sure though, it would help if you give me the structure docs for that JSON. What values are in the arrays on what indexes etc.

[Laukei]

Hi Henrik, sorry I took a while to reply! I have discussed this with Ash (@Sketchy502). The rendering method is presently being rewritten, so Ash is keen that we don't agree anything too concrete in case further information is needed.

I wrote a basic API auth/upload for the site a few weeks back, but it needs rewriting due to some misunderstandings I had when writing it, which I'll do once I've finished the current changes I'm mid-way through. When it's complete, it should be relatively easy for me to build some kind of parsing layer on top of that to take whatever the planner exports. If in the mean time you've developed full export-to-xml-savegame functionality we can take that, but if not we can work out what the planner can provide, what needs to be inferred, and what will hold up rendering without an explicit fix.

Does that sound like a reasonable route forward?

[hpeinar]

That sounds reasonable.

In the mean time I can build planner endpoint which upload.farm could use for linking. If I happen to have extra time, I'll look into exporting the xml.

I'll update this issue once I have something.

[Laukei]

Hi Henrik! We're almost there with the features we were hoping to add for the next iteration. If there is some way for us to pass save data into the planner we can try to include that too. Did you have a preferred method? I think we can also start looking at what needs doing to take the raw output from the planner and turn it into a picture.

[hpeinar]

Sorry for me being so slow.

I'm still trying to find time to finish off the XML parser on the server side to give you an endpoint. I hope to get it ready in few days, I'll certainly let you know then.

[hpeinar]

So... Almost a month has passed but I finally got the time to do the endpoint. (actually doing the endpoint wasn't a problem, it was moving the XML importer to server-side which took time).

So how does it work:

1. You get the saveGame.xml from the user.
2. You post it to the stardew.info/api/import
3. You'll get URL on which user can edit that saveGame file

You can read more about it from here: https://github.com/hpeinar/stardewplanner#integration

If you have any problems or need something changed, please let me know.

[Laukei]

Hey Henrik, thanks for the update! Looks well documented. I will look at integration in the coming week or so.

One thought I have is that we store uploaded saves zipped for efficiency (2mb XML becomes <100kb ZIP). If you're interested in keeping bandwidth usage down we can send this zip and you can unzip it locally.

All uploads are assigned an ID (e.g. 1AFQF1). We store the save for that as a zip, with filename 1AFQF1. Within the zip is one file, also named 1AFQF1, which is the XML.

Of course if you're happier just taking the straight XML, that's no problem, we can just unzip the file before sending!

Let me know if that's desirable for you, but otherwise I'll assume you just want the plain XML.

[hpeinar]

Ziping the xml makes perfect sense, I'll definitely implement it. I'll update once done (Not planning to take whole month this time ;))

[hpeinar]

Are you guys using zip format or would gzip also be an option?

[Laukei]

At present we're using zip with ZIP_DEFLATED compression from python's zipfile module (implementation here). What advantages would gzip offer? I'm open to switch if there's a benefit!

[hpeinar]

zip will work just fine, I just wanted to make sure I'll be implementing the correct format handling.

[hpeinar]

Alright, I've pushed the .zip implementation live. Let me know if you have any more needs / problems with the endpoint.

The usage didn't change, the endpoint just checks if incoming file is zip or not (by checking 2 first bytes of file header).

[hpeinar]

Moving on to the image creation we talked about earlier. Would it be sufficient to just generate the farm part of the XML? Without all the other meta that goes into the saveGame files. What would be the necessary parts of the XML your renderer needs to render the image?

I'm asking this because in the planner, there isn't always save game present, users can start to plan their farm from a "clean sheet" without uploading the save file.

[Laukei]

Hey Henrik, apologies for the lack of movement from us on your API, we've been a bit preoccupied with UK political events to have much free time!

We use a much smaller subset of the data from the XML than the full save. We store it as JSON so it should be relatively easy for us to take data from the planner as well. This is an example from this farm. The dictionary is grouped by type, and (in Python) each entry is a namedtuple (flattened to a list) with values: ['name', 'x', 'y', 'w', 'h', 'index', 'type', 'growth', 'flipped', 'orientation']

I've not looked at what is stored by the planner, but I'm imagining it's not wildly different from this? @Sketchy502 should probably join this conversation too, he wrote most of the rendering code.

[hpeinar]

Hei!

No worries about the integration, I'm aware of the political shenanigans going on in UK right now, so I fully understand.

Thanks for the JSON example, I'll see if I can make the planner to output something like that. Although the planner is lacking the following information:

• index (I presume this is the in-game asset index?)
• growth (Planner does not differentiate growth levels)
• flipped (Not sure what this would be?)
• orientation (Same problem, for the planner everything has the same orientation as far as roads and dirt etc goes)

I assume, apart from the index, I can use fixed values for now and try to improve them over-time (like the orientation).

[Sketchy502]

Hi Henrik,

I think the simplest way is for us to just parse the JSON you currently store, as you linked in a previous comment. We can then fill in any missing information on our end since a lot of it is superficial, such as flipped telling us to mirror the sprite. Also, if you implement any additional features at a later date that supply part of the missing information we can just modify our code to pull from your JSON file.

The one thing I am not sure about is how you differentiate between different types of road/fence in your saves.

[hpeinar]

They just have a (string) "type" property. So it's "stone_road" or "wooden_fence" etc.

I would be fine with converting them (or adding) game indexes for these.

[Sketchy502]

I have found some time to have a play about using your api and can easily render an image from the JSON. I've made a quick render using only grass for now, which you can see here

I think for now it will be easiest for the both of us if I handle the details on our end. I plan to redo the rendering code and once that is done we can work on something better.

How would you like to handle things like crops? As you don't store any crop data I can just make some at random or leave the tilled soil empty.

[hpeinar]

Oh, the grass looks sweet.

Probably just select 2 crops in fully grown state, one for regular, one for tilled. Something very distinguishable, like parsnip and grapes. If you want to spice it up, you could random it per image (per tile would probably be too messy).

Otherwise nice! I'm looking forward to the image rendering.

[Sketchy502]

It's taking a little longer than expected to get this together but I wanted to give you a progress report on the rendering so far: http://imgur.com/a/oVOI8

I've given crops and normal trees a random growth of either full or the stage before full growth. For crops this adds a nice break to the uniformity but I'm not so sure on trees. What would you prefer? Shall I set every to full growth, just the trees or leave it as is? Also, would you like the chance for normal trees to become mushroom trees?

[hpeinar]

I'd set the trees to full growth. It's fine for crops to be random but as full grown trees might change how people plan their farm, I think it's better to show them in their full form. Also no, I don't think mushroom trees would be necessary.

Progress looks great so far!

Let me know if I need to do something on the planner part.

[hpeinar]

Long time no talk.

Just wanted to check how's going and if our plan is still in the action.

[Laukei]

Hi Henrik,

Apologies for the lack of progress, life runs away with you sometimes! I will be away until late September so won't be able to do anything until then, but once I'm back we'll finish the integration properly. It's long overdue, and with v1.1 coming we'll have to make a big update anyway, so it makes sense. I think Sketchy has been working and not pushing commits for an API to render farms for stardew.info too, so it should be a short burst of concerted effort to tie it all together.

Apologies again for the delay!

[hpeinar]

No worries!

I am not particularly in a hurry with this anyway, just wanted to check the status on it :)

[hpeinar]

Just a heads up if you don't know yet, 1.1 beta has been released. http://community.playstarbound.com/threads/1-1-beta-thread.124827/page-13#post-3030604

[Sketchy502]

Thanks for the heads up on the beta. ConcernedApe announced that the update will go live on Monday (https://twitter.com/ConcernedApe/status/781573088529879040). Looks like it'll be a fun weekend.

Sorry for the lack of communication about creating renders, real life takes over from time to time. I have the rendering of plans from stardew.info working, we just have to create the API and find a nice way to integrate it our site.

[Laukei]

Hi Henrik, we've pushed our v1.1 update now (though it needs a couple of bug fixes) and I'm back on the upload.farm horse. I'm beginning work on the API tonight. I've not designed a proper API before, so fingers crossed it'll work okay!

[hpeinar]

Fingers crossed!

I hope you are going to use JSON 👍

[Laukei]

Hi again! I've completed the basic structure, but I wasn't sure how you wanted to handle certain things. So, in the most-basic form and running it locally, it currently behaves something like this:

PUT http://127.0.0.1:5000/api/v1/plan

Payload: { 'plan_json':{"buildings":[{"type":"stable","x":7... ...oad","x":656,"y":1024}]}, 'source_url':'https://stardew.info/planner/tough-horse-66/' }

Response: { 'status': 'success', 'url': 'http://127.0.0.1:5000/plan/1BUga9' }

So, first thing: I'm going to integrate the rendering code by @Sketchy502 later today or tomorrow, and work on a basic page to display the render and a link back to source_url so people can return to the planner from the render. Should the source_url be optional? I mean, if person 1 creates a design on the planner and person 2 gets the link, can they alter the design and overwrite it? That might be undesirable.

Second thing: we have user accounts on upload.farm which could conceivably 'claim' renders to a user account, and a separate voting system which could be used to rate them. I imagine that since you don't have user accounts it would realistically be pretty horrible to try to establish ownership over rendered farms so I should probably abandon that concept, but we could still implement a voting system to allow upload.farm registered users to upvote top designs. I've also noticed your URLs give quite nice names to farms, so I think that could perhaps become their autogenerated farm name. How does this sound?

Third thing: with upload.farm we check for duplicate uploads. Do you have some kind of unique identifier for each farm that could be passed to upload.farm, or only saved ones? Would you be reasonably content for us to just md5 the json and check for duplicate md5s?

Fourth thing: would you want the ability for your users to delete rendered plans? If we're unable to 'claim' farms to user accounts on upload.farm then realistically the only way to enable this is if we returned you a deletion key when you PUT the data, but I guess it's conceivable you have no way of linking that deletion key to a particular user.

Let me know!

[hpeinar]

Firstly: I'd suggest to change the PUT request to POST as per usual API standard POST is used to create a resource and PUT is used to edit a resource.

About the upload.farm -> planner integration: Planner currently does not check for duplicate uploads and just generates new url each time you upload your save (even if it is exactly the same). Also, every time you press save in the planner you get new url for your plan. So this means that the planner does not "update" the old one but rather creates a new one. Due to this behaviour the planner haven't had the need for accounts because you can't edit a plan once you have saved it and the link you get will always point to the same plan.

It does makes sense to check for duplicates and provide already built plan if you are using exactly the same data though, so this is something I'm going to implement. This means you could possibly tie a stardew.info url for each save upload in upload.farm. Though if you edit the plan you'll get new url but the url provided somewhere in upload.farm would always point to the exact version of the plan the save had at that moment.

So how I see the user flow:

1. User uploads his save to upload.farm
2. There would be a link or a button to "plan your farm"
3. When user clicks the button, upload.farm sends the save farmData to the planner and receives a URL which is then tied to the upload.farm page (this only needs to be done once for every save file upload).
4. Anyone (either the owner of the farm or someone else) can at any time click the planner link and will end up on the plan which features the layout the save had.

upload.farm Rendering API: For the rendering, I see this a totally independent thing from the upload.farm accounts. Users already get their farm render in upload.farm when they click on the image right, so if you'd like to have a voting of best (already made in game) farm plan, then you can use these.

The rendering API would be used by the planner to provide users with game-like pictures for the planned farms. I see this mainly be used like this:

1. User uploads his save at upload.farm
2. Clicks the plan your farm button
3. Has some crazy re-design plans and uses the planner to plan them out.
4. To confirm that these changes would also look pretty and desirable he then renders the picture as the planner does not currently offer game-like view.
5. He is happy with the plan and then carries out the changes in the game
6. He then again, uploads his new save at upload.farm etc.

If you feel like voting on different plans is something that should be done, this should probably happen on the planner side as this is where the planning actually happens. I would keep these two things apart: upload.farm for "real" farms (that people actually have in the game). stardew.info for "designed" farms (possibly even made by people who don't even own the game).

I hope all this makes some sense. I'm happy to continue this discussion if you have any more ideas / questions etc.

[Laukei]

Hello! I've changed the http method to POST as you suggested.

I have completed a first pass of the the upload.farm->stardew.info functionality (not live on upload.farm yet). It looks something like this:

The logic I've written sends the zip to stardew.info the first time this is requested and stores the link in the database so it doesn't need to re-send for future usage.

In the first instance I'm going to have this limited so only the original uploader can pass the farm to stardew.info. At the minute we haven't got a privacy policy for upload.farm and I'm wary of allowing third parties to send someone's savegame data to a fourth party!

Regarding the upload.farm rendering API: taking your feedback on-board and stripping the feature-creep back, I am now imagining something like this:

So a user would click on the 'render' button on stardew.info and would be redirected to a simple page on upload.farm with the image, maybe without the header bar even. Renders wouldn't be searchable and wouldn't show up on the front page. Is this agreeable?

[hpeinar]

This is totally agreeable and looks really good to me.

Did you test the zip sending to stardew.info, did it work fine with even new 1.1 saves?

[Laukei]

Happy to report that zip sending to stardew.info works great with saves from <1.1 and >=1.1!

I've begun looking at integrating the rendering code but I only have the link from https://github.com/Sketchy502/SDV-Summary/issues/7#issuecomment-216294759 to go off, and this is from v1.0 - could you give me a couple of links to json from farms rendered on the other maps? Also, I notice you have support for unofficial maps - @Sketchy502 what's your thoughts on whether we're able to support these?

[hpeinar]

Hey!

I'm glad to hear the zip part works nicely!

Regarding 1.1 json's https://stardew.info/planner/red-mouse-52/ => https://stardew.info/api/red-mouse-52 (fishing layout) https://stardew.info/planner/grumpy-snake-25/ => https://stardew.info/api/grumpy-snake-25 (regular layout with 1.1 buildings and lots of other stuff)

The layout is defined under options.layout and is currently in string format (regular, combat, fishing, foraging, mining). I can add the saveGame farm layout integer there too as this would probably make things easier on your side.

Regarding the custom layouts, the planner is indeed supporting them to some extent. I also have few waiting up to be added.

Let me know if you need anything!

[Laukei]

Hi Henrik,

I've been busy working on the API the last few days. It's 99% ready to go, but before we push it to upload.farm I wanted to check a couple of few things.

1. What does your API return if you're over the request limit? (I didn't want to hit your API 600 times in <15 minutes to try to find out experimentally, hehe)
2. When I've been using the upload.farm API in development I've been GETting examples from stardew.info/api/... as a string and passing them into the API as {'plan_json':'_string_of_stuff_I_got_fromurl'}. Would you prefer it if it was pure json?
3. We're looking at an initial API usage limit of 10 submissions that require renders (so not just retrieving a url from the db) every 60 seconds. I don't know how much traffic stardew.info gets and whether this is likely to be significantly too low or not - what do you think?

Cheers!

[hpeinar]

Hi again!

1. My API will return HTTP code 429 with message "Too many requests, please try again later." (non-json I'm afraid, rely on the HTTP status code) when you have bypassed the ratelimiter limits.
2. I would prefer if it was pure JSON indeed.
3. Planner is getting around 3-4k visits per day, I think it is fine if it's 10 per 60 seconds. Please do specify what your API returns if the rate limit is exceeded so I can let the users know accordingly should they hit it.

[Laukei]

Hi! API development and integration is done, I'll send another message when I've pushed it to upload.farm, but for reference:

POST http://upload.farm/api/v1/plan

Receives JSON-format input with Content-Type: application/json

Rate limit is currently 10 per 60 seconds. Maximum payload size is 16MB.

Expected format:

{
   "plan_json": {json},
   "source_url": "https://stardew.info/planner/id",
   "season": "season"
}

season is optional and is one of "spring", "summer", "fall", "winter", and sets the rendering theme. If not present, "spring" is default. If source_url is not on an approved list of domains (presently just stardew.info) then it's not displayed on upload.farm, to prevent malicious redirection to other URLs.

Response 200:

{
   "status": "success",
   "url": "http://upload.farm/plan/url"
}

Responses 400 or 429:

{
   "status": error_message
}

error_message is one of:

1. "no_json_header", can't find any JSON because the header isn't set
2. "bad_input", can't verify all the required elements are in the JSON
3. "over_rate_limit", API is over rate limit
4. "unsupported_map", map is not supported by upload.farm
5. "failed_conversion_to_local_structure", couldn't convert something in the input JSON for rendering

If a submission is made which has the same md5 as another submission and the season of the render is the same, the existing entry is returned to prevent duplication. Additionally, to prevent filling the server with 200kb image files, a limit of the 10,000 most recently viewed renders will be stored. After this point, if someone tries to view a link to a deleted render, they'll see a holding page and the server will regenerate it for them on demand. (Whether this is a smart design decision remains to be seen, but it was certainly fun to code!)

[hpeinar]

Thanks, I'll try to write my side of the rendering API integration today so it would be ready to go live once you push your changed to upload.farm.

[Laukei]

Oh, for reference we haven't added support for any unofficial maps yet, so rendering will only work for the official 5 at this point.

[hpeinar]

I got that from the possible error message yeah. It's fine, the amount of un-official maps are so small anyway, I'll probably revert these to the regular layout for the rendering for now.

[hpeinar]

It's going to look like this. The bottom link just takes to upload.farm homepage.

[Laukei]

Hi Henrik! I have updated the server now, you should be able to test the API out. I tried it myself here. Let me know if you run into anything unexpected.

I tested sending a few farms from upload.farm to stardew.info too, with two minor bugs, probably both with upload.farm:

1. An unexpected Python Requests SSLError - "stardew.info hostname does not match beta.stardew.info", which I never had when running locally, so I've suppressed the need for the SSL certificate to be verified. I'm not sure why this error only occurs on the server and not when running locally though, it's a bit strange.
2. Newer farms can be imported to stardew.info API with no problem, but older ones seem to just seize up. I'm not completely certain why this is. I'll have to test them out this evening on my own machine.

I like the powered by upload.farm!

[hpeinar]

So I've gotten it to semi-work, I still need to do some UX stuff before putting it live but I ran into some "failed_conversion_to_local_structure" errors.

Is it possible that your renderer will not render the picture if some of the items are placed into non-allowed places? (like on the lake for an example).

The one json I've gotten this error is here (don't have this live yet, that's why gist): https://gist.github.com/hpeinar/6b009fd11921a06798dd2d76e9ab95f9

[Laukei]

Hmm, the problem seems to be with twigs and stones. I don't know why, they worked on previous test renders. I'll try to patch that quickly.

[hpeinar]

Cool! No hurry, I won't put it live on my side before tomorrow.

I'll make a test json with every possible building and tile in it which the planner supports to test this later on.

[Laukei]

I was using this one in my testing: https://stardew.info/planner/ancient-elephant-79/ - it still renders for me, but a new plan with just a twig and stone in it doesn't.

Looking at the api output, 'twig' comes under 'building' in that - but in the JSON you linked to on Gist it's in 'tiles'. I think this might be the cause of the issue. Did you move it between categories in an update maybe? Anyway, it should be simple enough to fix for us!

[hpeinar]

Twigs and other little rocks should be tiles! It is currently wrong when you use them from the menu but it is correct when you import save. I'm going to fix this right away.

Would it be possible to ease the rendering up though? Maybe it could just ignore the ones it doesn't understand? The planner currently uses a placeholder [?] sprite for the items it doesn't understand, you can actually see it in your test save (just besides the stump)

Edit: Alright, I've fixed the twig placing and deployed the fix live. It now correctly goes under tiles as it should.

Edit2: Use this save for testing now: https://stardew.info/planner/slimy-turtle-85/