Alter JSON format

[WebBreacher]

I am throwing out the following potential changes to the JSON file format for discussion:

Shortening parameter names

With over 4000 lines of JSON, we need to optimize the data and one easy way to do that is to shorten our parameters. Moving from verbose params like account_existence_string to e_string saves use thousands of bytes. Do that for the other params, and we get a bit more efficient.

Standardizing the URI params

Also, I'd like to shift from using check_uri and pretty_uri to uri_check and uri_pretty (moving the URI to the beginning) for consistency.

Adding HTTP POST support

Per #402 and input from @janbinx, we would add post_body param which, if it contained data, then an HTTP POST would need to be sent with the data in the field. Otherwise, with a blank/empty post_body field, an HTTP GET would be sent.

HTTP Response Codes change from strings to integers

We will be changing the e_code and m_code from strings (surrounded by quotes) to integers.

Example entry would look like the below:

Current entry:

    {
       "name" : "diigo",
       "check_uri" : "https://www.diigo.com/interact_api/load_profile_info?name={account}",
       "pretty_uri" : "https://www.diigo.com/profile/{account}",
       "account_existence_code" : "200",
       "account_existence_string" : "regist_at",
       "account_missing_string" : "{}",
       "account_missing_code" : "200",
       "known_accounts" : ["whoami", "johndoe"],
       "category" : "images",
       "valid" : true
     },

Proposed HTTP GET entry:

     {  
       "name" : "Example GET",
       "uri_check" : "https://www.example.com/load_profile_info.php?name={account}",
       "uri_pretty" : "https://www.diigo.com/profile/{account}",
       "post_body" : "",
       "e_code" : 200,
       "e_string" : "regist_at",
       "m_code" : 200,
       "m_string" : "Account not found",
       "known" : ["whoami", "johndoe"],
       "cat" : "images",
       "valid" : true
     },

Proposed HTTP POST entry:

     {  
       "name" : "Example POST",
       "uri_check" : "https://www.example.com/interact_api/load_profile_info.php",
       "post_body" : "Name=Gareth+Wylie&Age=24&Formula=a%2Bb+%3D%3D+21",
       "e_code" : 200,
       "e_string" : "regist_at",
       "m_code" : 200,
       "m_string" : "Account not found",
       "known" : ["whoami", "johndoe"],
       "cat" : "images",
       "valid" : true
     },

Migration

We will continue to maintain both the old and new formats of the JSON files for a month or so while tools adjust to the new format. Old JSON will continue with the current name. New format of the JSON will be named wmn-data.json.

Comments

Thoughts? Put your comments below. Looking to close discussions and solidify changes by 22 July.

[janldk]

If saving space is a goal, why not remove the get parameter completely and instead just instruct tools that use the json-file to perform a GET unless post_body is present? That would save a considerable amount of characters.

[WebBreacher]

@janbinx that makes a lot of sense. The post_body field would act as a Boolean true/false field as it'll have a value (and require a POST) or not and be a GET.

[Unpublished]

My thoughts while assuming size for downstream projects like https://whatsmyname.app is your concern.

I'm in favor of providing a separate minified json file. Minifying shrinks size more than shortening parameters.

Minified example: {"name":"Example GET","uri_check":"https://www.example.com/load_profile_info.php?name={account}","uri_pretty":"https://www.diigo.com/profile/{account}","post_body":"","e_code":"200","e_string":"regist_at","m_code":"200","m_string":"Account not found","known":["whoami","johndoe"],"cat":"images","valid":true},

Minifying while keeping human-readable parameters sounds like a good compromise/solution to me. The minified version could be automatically generated via github actions -> Only the human-readable version needs to be maintained.

Regarding post_body (https://github.com/WebBreacher/WhatsMyName/issues/402#issuecomment-1155400744):

I'd rather introduce a request_method param which defaults to GET and a request_body param. This way (rare?) edge cases like POST with empty body and GET with body are also covered.

It'd be more future-proof if a site uses HTTP OPTIONS method. Altough I only saw OPTIONS requests followed by a GET/POST requests, yet.

[WebBreacher]

Thanks for the discussion everyone.

@Unpublished - While I understand your point about the request_method parameter and value, I think I'm going to go with @janbinx suggestion of not using a request method param. If there is data in the post_body param then it is a POST, otherwise a GET.

@Unpublished - the minify idea is excellent and I'll make an action for this (or, if you know how, feel free to suggest the code here). I started making one https://github.com/WebBreacher/WhatsMyName/blob/main/.github/workflows/minify-json.yml but it does not write the finished minified JSON to the project yet.

Moving Forward

I will be making the following changes to move forward on this issue:

1. Make new JSON file called wmn-data.json
2. Search and replace:
   1. check_uri → uri_check
   2. pretty_uri → uri_pretty
   3. account_existence_code → e_code
   4. account_existence_string → e_string
   5. account_missing_string → m_string
   6. account_missing_code → m_code
   7. known_accounts → known
   8. category → cat
3. Add "post_body" : "", line above "e_code"
4. Alter the 2 Python checker scripts to use new file and format

[WebBreacher]

PR #477 contains the wmn-data.json file in the new format.

TODO -

1. Update checker scripts to use it
2. Update JSON validator GitHub Action

[WebBreacher]

PR #478 Adds the new JSON schema file and updates the JSON validator GitHub Action

[WebBreacher]

@Unpublished - GitHub Action to minify code is in place and auto-generating the wmn-data-min.json and web_accounts_list-min.json files per your suggestion.

[yooper]

Can we get the status codes to be integers, and not strings? Otherwise, we have to cast within the python script.

"e_code":"200" -> "e_code":200

[WebBreacher]

Not sure how I/we missed that. Yes both m_code and e_code can/will be changed to integers instead of strings.

[WebBreacher]

Commit https://github.com/WebBreacher/WhatsMyName/commit/c609d0ab6d09c8650d2fb1a436c4e9454bd760ac addresses this @yooper

[jocejocejoe]

Should new pull requests modify both web_accounts_list.json and wmn-data.json for now?

[WebBreacher]

Should new pull requests modify both web_accounts_list.json and wmn-data.json for now?

Yes.

[WebBreacher]

As shown above, I inserted issues into all the project that are known to use our JSON.

Also, the https://whatsmyname.app tool has already been converted to use the newly formatted JSON.

[WebBreacher]

Notified the internet through our Twitter account about transitioning to the new format as of 1 Sep. Old file will still be here but will be unmaintained (nothing new/fixed). Tweet https://twitter.com/whatsmynameproj/status/1564304387111100416

[WebBreacher]

Removed old JSON from GitHub Actions/Checks

[WebBreacher]

Switched documentation to refer to the new JSON file as the "current" one and the old one as, well, "old".