JSON Formatting Help Needed

jeroenbrussich commented 3 years ago

we should really try to come up with some guides about JSON-files for Improved Initiative.

I am trying to understand how to create said JsonFiles but I can't seem to find any consistency in them. Allow me to explain.

upload

I found (this file) online with working data for Improved Initiative. If you would open the file, you would see about 20 "oneliners" with incorrectly formatted json data. Paste the content in https://jsoneditoronline.org/ if you would like to see what I mean.

download

Anyway, the file works for II. So I import the data and download it immediately. That gives me 02 - download initial upload.txt. Open this file and you will have incorrectly formatted _"ImprovedInitiative.PersistentCharacters"-oneliners and correctly JSON-formatted "Creatures"-blocks. Again, paste the content in https://jsoneditoronline.org and scroll all they way down to see what I mean.

about creatures

When I strip one creature from the first file, the one with the incorrectly formatted oneliner, I can upload that data if the array "ImprovedInitiative.Creatures": "[\"j1van1rm\"]" is present.

When I strip one creature from the second file, the one I downloaded with some correctly formatted JSON data, I can successfully upload correctly formatted json-data. The "array" does not even have to be present, I just have to start my data with "Creatures.

Just for trying, I combined the array with the JSON-data, but that fails..

about characters

When I strip one creature from the first file, the one with the incorrectly formatted oneliner, I can upload that character if the array "ImprovedInitiative.PersistentCharacters": "[\"mcerm30e\"]" is present.

HOWEVER, when I convert that file to valid JSON data , and omit said array, nothing happens. I tested with both "PersistentCharacters." and "Characters. but my Characters-tab remains empty when I serve this file.

So, my big question is:

how do I need to serve character-data to Improved Initiative if I want to keep the data valid JSON?
What am I doing wrong that my valid JSON data is not creating a character?

cynicaloptimist commented 3 years ago

Hey there,

Thanks for the in-depth notes. The JSON files aren't really designed to be user-editable. It's been a long time since I touched the code that deals with these but it's mostly in this file: https://github.com/cynicaloptimist/improved-initiative/blob/development/client/Utility/LegacySynchronousLocalStore.ts

The JSON files are produced by JSON.stringify, so it would be strange if they were incorrectly formatted. See this method: https://github.com/cynicaloptimist/improved-initiative/blob/development/client/Utility/LegacySynchronousLocalStore.ts#L92

The one-liners you're referring to are valid JSON. Each of them is an index of the keys of the items in the file. This is required for localstorage interoperability.

I have no plans to make these files more user-editable, but I am currently, slowly, working on a Library Manager UI that will include things like bulk export (so you can pick and choose what to include in a JSON file.)

Hope that helps!

jeroenbrussich commented 3 years ago

Hi there @cynicaloptimist

Thank you for your response! I understand that you don't want to touch working code :) If it ain't broken, don't fix it

Can I just add 2 small remarks before you go and close this "issue"?

When I say that the "ONELINER" is no valid JSON, I mean that no JSON-parser can parse a correct tree, probably because of the \" you can see on each line..

I fiddled around in python and got these results.

the oneliner has \"
if I change \" for ", I can parse that line into a json-tree [= json.dumps( var, indent=4)]. Pay attention to the double quotes without the backslash.
if, however, (and don't ask my why :) ), i parse the json data a second time [= json.dumps(json.dumps(var)), I get the "incorrect" json that your app also returns, meaning the \" is back again

So while you are correct I was a little bit harsh in my wording to call your data invalid JSON, it is however strangely formatted, as if the data was twice exported to json And I don't think that is because of json.stringify() because the testpage returns (what I call) valid json

all of point 1 does of course not matter, because your app works as intended!

However, it is kind of strange that my "valid" json is not accepted by the importer, and the "invalid" json is, but only for characters, not for creatures . So maybe, just maybe, there is something going on there? I have too little knowledge of node.js to help you look into it. But maybe it is worth looking into in the far future?

For the time being, you may close this issue as being resolved :D

afbeelding

japrozs commented 3 years ago

trying using JSON.parse on the data

shakefu commented 3 years ago

oneliner in that case is a valid JSON string, which includes escaped double-quotes \" when printed as a string value. json.dumps takes a dict (or other JSON compatible type) and encodes it. So your red highlighted code is in fact double-encoded.

The valid Python would look like:

import json, pprint
oneliner = "{\"oneliner\": true}"  # String with JSON encoded data
data = json.loads(oneliner)  # Parse the string into Python types
pprint.pprint(data)   # Print the Python data structure nicely
nice = json.dumps(data, indent=4)  # Dump the Python data into a nice human readable JSON encoded string
print(nice)  # Print the string containing nicely formatted data

japrozs commented 3 years ago

You can try doing the following:

const obj = {\"foo\" : 3}
const str = obj.toString();
str.replace("\"", '"');
obj = JSON.parse(str)

cynicaloptimist / improved-initiative