Switch to compact pretty JSON

[nyurik]

JSON is notoriously bad with versioning, especially the large files. One way to make it a bit easier to diff is to use a "compact pretty" form of JSON.

Unfortunately, I only found a javascript (node-js) library https://stackleap.io/js/json-stringify-pretty-compact -- the python one is apparently missing. I am ok to use whichever lang, and at this stage it would be fairly trivial to rewrite the one utility in JS (despite my dislike for the language). It would make it much easier to work with the raw json though.

P.S. When comparing it locally, I use VS Code plugin to prettify JSON -- makes it far easier to compare locally.

[bcwhite-code]

When exporting from Factorio, the order of things, like electrical connections, can change each time. At least with non-compact, the diff is fairly small, line-wise, rather than showing the entire entity (single line) having been changed.

Granted, that doesn't make any difference to git which uses binary diffs (not line diffs) for its storage.

Still, I expect that if I were to recapture a blueprint, especially if rotated, then the diff will be huge either way.

On Fri, Jun 24, 2022 at 7:38 PM Yuri Astrakhan @.***> wrote:

JSON is notoriously bad with versioning, especially the large files. One way to make it a bit easier to diff is to use a "compact pretty" form of JSON.

Unfortunately, I only found a javascript (node-js) library https://stackleap.io/js/json-stringify-pretty-compact -- the python one https://stackoverflow.com/questions/45681702/compact-but-pretty-json-output-in-python is apparently missing. I am ok to use whichever lang, and at this stage it would be fairly trivial to rewrite the one utility in JS (despite my dislike for the language). It would make it much easier to work with the raw json though.

P.S. When comparing it locally, I use VS Code plugin https://marketplace.visualstudio.com/items?itemName=Kyle.vscode-pretty-compact to prettify JSON -- makes it far easier to compare locally.

— Reply to this email directly, view it on GitHub https://github.com/bcwhite-code/brians-blueprints/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLEM3CQOWBAX3IDP76WPYLVQZBHXANCNFSM5ZZI6ADA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Brian @.***

Treat someone as they are and they will remain that way.Treat someone as they can be and they will become that way.

[nyurik]

I agree about massive changes like rotating would cause a havoc. I wonder if entity ordering is important though -- could we simply re-sort the list of entities according to the (x,y) position when storing, and sort it by the entity number when encoding?

Often the pretty compact form would not show the entire entry in one line, but rather it would show some sub-values as more compact:

{"entity_number": 669, "name": "straight-rail", "position": {"x": -67, "y": -145}},
{"entity_number": 670, "name": "logistic-train-stop-output", "position": {"x": -65.5, "y": -145.5}},
{
    "connections": {"1": {"green": [{"circuit_id": 1, "entity_id": 897}], "red": [{"entity_id": 699}]}},
    "control_behavior": {"circuit_condition": {"comparator": ">", "constant": 0, "first_signal": {"name": "signal-anything", "type": "virtual"}}, "use_colors": true},
    "entity_number": 671,
    "name": "logistic-train-stop-input",
    "position": {"x": -64.5, "y": -145.5}
},
{
    "color": {"a": 0.49803921580314636, "b": 1, "g": 0.501960813999176, "r": 0},
    "connections": {"1": {"green": [{"entity_id": 699}]}},
    "control_behavior": {"read_from_train": true, "train_stopped_signal": {"name": "signal-T", "type": "virtual"}},
    "entity_number": 672,
    "name": "logistic-train-stop",
    "position": {"x": -65, "y": -145},
    "station": "[item=coal]8 ⇝ [item=lab]   Coal @ 1kSPM"
},
{"entity_number": 673, "name": "straight-rail", "position": {"x": -47, "y": -145}},

[nyurik]

P.S. In the same spirit of simplifying things, I think the metadata files just add lots of noise without being very useful - perhaps it would be better to have one file per dir that acts as a catalog, possibly with a simple script (part of the .github/workflows CI) that validates that all json files are listed in it.

[bcwhite-code]

I'm sure the tool could be improved to make for smaller diffs but I really don't think it's worth it. All told, it's a very small amount of data and what's important is more the ability to "undo" than to compare changes.

The .metadata file alongside the .json one is information about how the blueprint fits into the book and is currently only its index number. I did toy with the idea of making the index part of the filename but this seemed more future-proof.

On Fri, Jun 24, 2022 at 8:09 PM Yuri Astrakhan @.***> wrote:

I agree about massive changes like rotating would cause a havoc. I wonder if entity ordering is important though -- could we simply re-sort the list of entities according to the (x,y) position when storing, and sort it by the entity number when encoding?

Often the pretty compact form would not show the entire entry in one line, but rather it would show some sub-values as more compact:

{"entity_number": 669, "name": "straight-rail", "position": {"x": -67, "y": -145}},

{"entity_number": 670, "name": "logistic-train-stop-output", "position": {"x": -65.5, "y": -145.5}},

{

"connections": {"1": {"green": [{"circuit_id": 1, "entity_id": 897}], "red": [{"entity_id": 699}]}},

"control_behavior": {"circuit_condition": {"comparator": ">", "constant": 0, "first_signal": {"name": "signal-anything", "type": "virtual"}}, "use_colors": true},

"entity_number": 671,

"name": "logistic-train-stop-input",

"position": {"x": -64.5, "y": -145.5}

},

{

"color": {"a": 0.49803921580314636, "b": 1, "g": 0.501960813999176, "r": 0},

"connections": {"1": {"green": [{"entity_id": 699}]}},

"control_behavior": {"read_from_train": true, "train_stopped_signal": {"name": "signal-T", "type": "virtual"}},

"entity_number": 672,

"name": "logistic-train-stop",

"position": {"x": -65, "y": -145},

"station": "[item=coal]8 ⇝ [item=lab]   Coal @ 1kSPM"

},

{"entity_number": 673, "name": "straight-rail", "position": {"x": -47, "y": -145}},

— Reply to this email directly, view it on GitHub https://github.com/bcwhite-code/brians-blueprints/issues/4#issuecomment-1166147011, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLEM3DHOTHEA4CCIBRA3H3VQZE2NANCNFSM5ZZI6ADA . You are receiving this because you commented.Message ID: @.***>

-- Brian @.***

Treat someone as they are and they will remain that way.Treat someone as they can be and they will become that way.

[nyurik]

Obsoleted by #8