Edit Raw JSON hangs on large save file

[Gravnaut]

Starting with 1.13.1 the save editor would freeze when clicking on "Edit Raw JSON" with a 2,491,193 bytes save file. I can open smaller save files. I switched back to 1.12.0 and had no issues. Windows 10, Java 8.0.3610.9

[Devilin-Pixy]

Might be useful to provide the log or a screenshot of the CMD window, to see if any issue shows. Also make sure to be using 64-bit Java.

[Gravnaut]

I’m attaching the save file that is causing issues and NMSSaveEditor.log

The program will eventually unfreeze but the editor does not appear. I can use the tabs for this save file, just not the JSON editor.

From: Devilin Pixy Sent: Monday, April 24, 2023 12:26 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Author Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

Might be useful to provide the log or a screenshot of the CMD window, to see if any issue shows. Also make sure to be using 64-bit Java.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[Devilin-Pixy]

@Gravnaut I am not seeing your attached save or the log, which might be due to response through email.

I have however just found a save that may have similar/same issue. Everything appears to work fine, until you try to open the Raw JSON in the editor. As soon as I do, it stops doing anything, but nothing showing in the output/log either. In my case it is a previous Expedition save from Utopia.

I am currently investigating. Just going in-game to save freshly again does not fix the issue. If I warp to a different system in-game and then save, the issue gets fixed. The system I was in is busy with bases of course, due to having been an Expedition system. Might be a relation to the issue. Nothing else solid just yet. Currently comparing the save from prior to warp, to the one after.

Possibly related issue #796

Hopefully I learn more, or @goatfungus has a clue.

[Gravnaut]

Your description of the hang is identical to what I was experiencing.

I uploaded both of my slot 2 save files and a copy of the Log here.

http://gravnaut.com/NMSSaveEditor.log http://gravnaut.com/save3.hg http://gravnaut.com/save4.hg

I’ve appreciated all the work you’ve done for NMS. I find your save editor to be the most useful tool I’ve used in a long time. Until money got tight I was a patron under the name of Jack Harper @.***).

Thanks, Robert Jacobson

From: Devilin Pixy Sent: Monday, April 24, 2023 1:23 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

@Gravnaut I am not seeing your attached save or the log, which might be due to response through email.

I have however just found a save that may have similar/same issue. Everything appears to work fine, until you try to open the Raw JSON in the editor. As soon as I do, it stops doing anything, but nothing showing in the output/log either. In my case it is a previous Expedition save from Utopia.

I am currently investigating. Just going in-game to save freshly save again does not fix the issue. If I warp to a different system in-game and then save, the issue gets fixed. The system I was in is busy with bases of course, due to having been an Expedition system. Might be a relation to the issue. Nothing else solid just yet. Currently comparing the save from prior to warp, to the one after.

Possibly related issue #796

Hopefully I learn more, or @goatfungus has a clue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Gravnaut]

The number of bases in PersistentPlayerBases is what I suspected as a cause. There are a total of 128 bases (all my bases, no other player bases) in the save file.

From: Devilin Pixy Sent: Monday, April 24, 2023 1:23 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797) The system I was in is busy with bases of course, due to having been an Expedition system. Might be a relation to the issue. Nothing else solid just yet. Currently comparing the save from prior to warp, to the one after.

[Devilin-Pixy]

In my own save I am testing with, I am in the Utopia star system, with my save data containing 100 bases (limit for 'other bases' I believe). With using the previous version of NMSSE (1.12.0), only removing those bases, the save loads fine again in 1.13.3 JSON editor. So I have certainly come to the same conclusion for that matter.

However, there must be more to it, as 1.12.0 does not have this issue and can load the same save without issue, showing all 100 bases as well in the JSON editor. Something with the updates has changed since 1.12.0 to cause an issue with base loading in the JSON editor, where it 'hangs' in 1.13.3 in case of a large amount of bases.

Now testing version 1.13.2 shows the issue to also be present, so that at least narrows down another update version. I do not have versions 1.13.0, or 1.13.1 to test.

I am leaving my testing at this for now. I think there is enough info to work with.

My Utopia slot 15: Issue save.zip

[Gravnaut]

For my save the issue began with 1.13.1 when I updated the program from 1.12.0

Another change NMS made to base parts is that to EVERY base part there is now a message line appended.

,"Message":""

Whether that has any bearing on the issue is another matter, but it seems odd for the game to be appending that variable to every base part.

From: Devilin Pixy Sent: Monday, April 24, 2023 2:59 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

In my own save I am testing with, I am in the Utopia star system, with my save data containing 100 bases (limit for 'other bases' I believe). With using the previous version of NMSSE (1.12.0), only removing those bases, the save loads fine again in 1.13.3. So U have certainly come to the same conclusion for that matter.

However, there must be more to it, as 1.12.0 does not have this issue and can load the same save without issue, showing all 100 bases as well. Something with the updates has changed since 1.12.0 to cause an issue with base loading, where it 'hangs' in 1.13.3 in case of a large amount of bases.

Now testing version 1.13.2 shows the issue to also be present, so that at least narrows down another update version. I do not have versions 1.13.0, or 1.13.1 to test.

I am leaving my testing at this for now. I think there is enough info to work with.

My Utopia slot 15: Issue save.zip

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Devilin-Pixy]

Oh wow, I had not noticed this, but after checking a save from February 4th, it only contained the mapping for "Message" (sHl) 16 times. Saving in-game and verifying the change, now has it count the same "Message" (sHl), 7750 times. That is insane! No doubt likely mostly present in base data, but still. I agree I am unsure why this would be needed ...

I can also ensure this is not Save Editor doing this, as I used a different tool to rule this out, only decompressing the save data and counting the exact mapping in both cases.

[Gravnaut]

Experimenting with the appended ,“Message”:”” base part description, I switched to 1.12.0 and copied all the data from PersistentPlayerBases, fed it into a parsing program to remove all the tabs and such, opened notepad++ and removed all 11,960 occurances of that variable, reinserted it into the save file and saved. I then switched to 1.13.3 and tried again and it still hangs, but the program does finally decide to quit trying to open the file on it’s own after about a minute. Evidently the “Message” part description has nothing to do with the error.

I then tried opening your save file and yours hangs also, but yours does not end on it’s own. I let it run for about 4 minutes and then used task manager to end the task. Both yours and my save files bring the CPU activity to 100% and essentially stays there the entire time.

This isn’t a do or die situation for me. Take what ever time you need to fix any other issues that have arisen. I just wanted to pass along the issue and wasn’t expecting any quick solution.

Thanks!

From: Devilin Pixy Sent: Monday, April 24, 2023 2:59 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

In my own save I am testing with, I am in the Utopia star system, with my save data containing 100 bases (limit for 'other bases' I believe). With using the previous version of NMSSE (1.12.0), only removing those bases, the save loads fine again in 1.13.3. So U have certainly come to the same conclusion for that matter.

However, there must be more to it, as 1.12.0 does not have this issue and can load the same save without issue, showing all 100 bases as well. Something with the updates has changed since 1.12.0 to cause an issue with base loading, where it 'hangs' in 1.13.3 in case of a large amount of bases.

Now testing version 1.13.2 shows the issue to also be present, so that at least narrows down another update version. I do not have versions 1.13.0, or 1.13.1 to test.

I am leaving my testing at this for now. I think there is enough info to work with.

My Utopia slot 15: Issue save.zip

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Gravnaut]

You’ve gotten my son interested in writing a program to capture planet info and because of that he’s been having issues with that same save file last week. He seems to have narrowed it down to the encoding used. This is a comment from our discord discussing it.

“Best I can tell is the encoding changed from late '21 to today. I tried my encoding detection script on my 12/23/21 save and it detected utf-8, it still fails with a similar error. On the saves that didn't work for you I detect Windows-1252, this is the same as the current but it doesn't like something about it. This tells me that those files aren't truly Windows-1252. I can get it to work if I switch to utf-8 and ignore errors. I'm thinking that its doing a bit of a guess when detecting encoding. I'm going to search for an alternate library that maybe is more accurate.”

Not truly Windows-1252?

Oh, for the good old days before Frontiers and the steam encoding!!!!

From: Devilin Pixy Sent: Monday, April 24, 2023 5:45 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

Oh wow, I had not noticed this, but after checking a save from February 4th, it only contained the mapping for "Message" (sHl) 16 times. Saving in-game and verifying the change, now has it count the same "Message" (sHl), 7750 times. That is insane! No doubt likely mostly present in base data, but still. I agree I am unsure why this would be needed ...

I can also ensure this is not Save Editor doing this, as I used a different tool to rule this out, only decompressing the save data and counting the exact mapping in both cases.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Devilin-Pixy]

A bit off topic for the issue here, but what your son was likely seeing was a change to the save format that happened with the Frontiers update (September 2021). Since then, the save files use compression, instead of the plain text approach we used to see with the JSON data. Hence why the file encoding changed from UTF-8 to what it now sees as ANSI. More info on the compression format can be found in a topic on Reddit: https://www.reddit.com/r/NoMansSkyMods/comments/phyird/not_an_actual_editor_new_save_format_simple_specs/ The tool I mentioned to have used in my previous post does exactly that. To check encoding of a file, open it in Notepad and check what it says when you select to 'save as', it usually comes out correct.

[goatfungus]

Regarding the encoding detection, the raw save files are in NEITHER utf-8 or windows-1252. They are a hybrid of utf-8 with raw binary intermingled in it, so not typical of any normal JSON output.

Prior to the editor version 1.13.0, all save files were decoded in windows-1252 format because it could not be parsed as utf-8 with those invalid binary characters present, which meant some unicode characters (chinese, etc) were just lost. When these values were written back to the save file they were technically being "corrupted" by the editor, but the game didn't seem to care.

The recent update from 1.12.0 -> 1.13.0 was a major overhaul of how the editor reads/writes the save files, and I was finally able to fully decode utf-8 characters AND the weird binary strings that the game produces in its output format. You'll see if you export the save file as plain JSON text that some strings now contain \xFF encoded characters, this is where the editor detected binary instead of a real character, and it will happily import that back in.

With this update, when the editor reads in a file containing both utf-8 and binary data and you save the file without making changes, the output is now identical to the original file, so nothing is lost.

[goatfungus]

As for the issue with the JSON Editor, it is possible that the output text representation of your save file is now a different size to what it was prior to 1.13.0, but it won't be by very much. You can try using the Export JSON method on both versions and see how they compare. It could be that the old version was close to some arbitrary threshold and the new version goes over it.

[goatfungus]

I've just confirmed on a large save file of my own that the editor was previously not placing spaces between named values in the JSON editor, so the new version does inflate the size by about 7-8%. It's amazing how the addition of one character to make the format easier to read can create such a difference!

I can probably remove the extra space pretty easily, but this will not completely solve the issue, as some saves will still grow too large for the buffers in the text field. I still have a few ideas on how i can fix it permanently, but it's not easy.

[Gravnaut]

I exported the same file using the three versions I now have. These are the results.

Exported JSON file sizes for save4.hg: 1.13.4 value 21,334,480 bytes 1.13.3 value 21,334,480 bytes 1.12.0 value 19,664,968 bytes

From: Brendon Matthews (GoatFungus) Sent: Monday, April 24, 2023 6:51 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

As for the issue with the JSON Editor, it is possible that the output text representation of your save file is now a different size to what it was prior to 1.13.0, but it won't be by very much. You can try using the Export JSON method on both versions and see how they compare. It could be that the old version was close to some arbitrary threshold and the new version goes over it.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Gravnaut]

Perhaps changing “Message” to “Msg” in the variables dictionary can also help? That one line in and of itself creates an additional 155480 bytes in my save file. Shortening it to “Msg” would reduce it to 107640 bytes for the resulting new line. Not a whole lot of savings, but...

From: Brendon Matthews (GoatFungus) Sent: Monday, April 24, 2023 7:03 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

I've just confirmed on a large save file of my own that the editor was previously not placing spaces between named values in the JSON editor, so the new version does inflate the size by about 7-8%. It's amazing how the addition of one character to make the format easier to read can create such a difference!

I can probably remove the extra space pretty easily, but this will not completely solve the issue, as some saves will still grow too large for the buffers in the text field. I still have a few ideas on how i can fix it permanently, but it's not easy.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[goatfungus]

I have figured out the issue with the JSON Editor now... it was to do with the support for unicode characters as well as the inflated output size. I have fixed this in the latest version 1.13.5.

[Gravnaut]

Thanks so much. I can confirm that it works. It’s been a pleasure!

From: Brendon Matthews (GoatFungus) Sent: Monday, April 24, 2023 10:23 PM To: goatfungus/NMSSaveEditor Cc: Gravnaut ; Mention Subject: Re: [goatfungus/NMSSaveEditor] Edit Raw JSON hangs on large save file (Issue #797)

I have figured out the issue with the JSON Editor now... it was to do with the support for unicode characters as well as the inflated output size. I have fixed this in the latest version 1.13.5.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>