search

aazuspan / snazzy

✨🗺️ Snazzy basemaps and Font Awesome icons in the Earth Engine code editor
31 stars 3 forks source link

JSON parsing error #2

Closed aazuspan closed 2 years ago

aazuspan commented 2 years ago

The Bright & Bubbly style causes a JSON parsing error. This is the only style I've seen that fails, but there may be others.

To recreate:

snazzy.addStyle("https://snazzymaps.com/style/17/bright-and-bubbly");

Error:

Line 25: JSON.parse: unexpected character at line 1 column 1730 of the JSON data
aazuspan commented 2 years ago

If you look at the json field for that style, it contains "comments" that are not valid JSON. For example:

[{"color":"#efe9e4"},{"lightness":-10}]},//-----HideslabelsforPointsofInterests-----

To make that style work, those comments will need to be parsed and removed. Other styles should be searched for similar patterns that would break JSON parsing.

aazuspan commented 2 years ago

Running json.loads on all of the styles fails with 5 styles:

Next step is to diagnose why the other styles are failing (they don't include // like Bright & Bubbly) and see if they can be repaired automatically. If not, they may need to be dropped (not bad to lose 5 out of 23k styles). Also worth noting that these problem styles are within the first 150 styles added, so my guess is that Snazzy Maps now applies more effective data cleaning and this problem shouldn't arise if new styles are added.

Still, it will be worth incorporating a json.loads test into the data ingestion script to ensure new styles don't break.

aazuspan commented 2 years ago

Just finished diagnosing. The issues are (in order above):

  1. Trailing commas
  2. Trailing commas
  3. Property names not in quotes
  4. Property names not in quotes
  5. C-style comments

Trailing commas and property names can be fixed by the dirtyjson package, but I'm hesitant to rely on that because it's not maintained. ast.literal_eval can only fix trailing commas, not property names. Haven't found a solution for C-style comments yet, but probably regex :vomiting_face:

aazuspan commented 2 years ago

All 5 broken styles have been fixed with the newly ingested asset.

The updating script has been modified to automatically fix the existing issues and validate all JSON prior to CSV export (14c80b20acb4701d47d8a1f9738fc6605877b268).