Improve handling of nested JSON responses

elvarb commented 3 weeks ago

Is your feature request related to a problem? Please describe. As it stands the plugin allows to iterate through the response json if it contains an array. But if the response json contains multiple arrays or a mix of arrays and normal values you can not get data from both.

For example the response from the urlscan.io search api

https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1

Returns

{
  "results": [
    {
      "submitter": {},
      "task": {
        "visibility": "public",
        "method": "api",
        "domain": "click.m.voacountrymusicfest.com",
        "apexDomain": "voacountrymusicfest.com",
        "time": "2024-08-23T11:47:49.143Z",
        "uuid": "d3b24120-c015-48da-804f-7591688982fb",
        "url": "https://click.m.voacountrymusicfest.com/f/a/CZ6N0ZGhP20dMAACkeXbug~~/AAJOsQA~/RgRolM1LP0UCMzJE82h0dHBzOi8vYXBwLmhpdmUuY28vZW1haWwvZWx0Lz9oX3NpZD0wZGQ3ZmE1ZGY5LTY2ZDBiZGFhYWE5ZjBmMDI3ZTU1MGFmOCZoYXNoPWY4ZGNlY2QxZTBhYmEwOCZuZXh0PWh0dHBzJTNBLy92b2Fjb3VudHJ5bXVzaWNmZXN0LmNvbS9hY2Nv...%20311%20...lAY29sbGFiLnNvY2lhbFgEAAAAAA~~"
      },
      "stats": {
        "uniqIPs": 8,
        "uniqCountries": 1,
        "dataLength": 2782471,
        "encodedDataLength": 980929,
        "requests": 42
      },
      "page": {
        "country": "US",
        "server": "gws",
        "redirected": "off-domain",
        "ip": "2607:f8b0:4006:820::2004",
        "mimeType": "text/html",
        "title": "Google",
        "url": "https://www.google.com/",
        "tlsValidDays": 83,
        "tlsAgeDays": 23,
        "tlsValidFrom": "2024-07-30T12:32:53.000Z",
        "domain": "www.google.com",
        "umbrellaRank": 10,
        "apexDomain": "google.com",
        "asnname": "GOOGLE, US",
        "asn": "AS15169",
        "tlsIssuer": "WR2",
        "status": "200"
      },
      "_id": "d3b24120-c015-48da-804f-7591688982fb",
      "_score": null,
      "sort": [
        1724413669143,
        "d3b24120-c015-48da-804f-7591688982fb"
      ],
      "result": "https://urlscan.io/api/v1/result/d3b24120-c015-48da-804f-7591688982fb/",
      "screenshot": "https://urlscan.io/screenshots/d3b24120-c015-48da-804f-7591688982fb.png"
    }
  ],
  "total": 10000,
  "took": 97,
  "has_more": true
}

If I want to get the information on the scan date and details from the page I can not do both as the show function only allows for one {..} iteration at a time meaning to get the information I want I have to have three requests to the API.

url: https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1
response-type: json
show: results -> {..} -> task -> time

url: https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1
response-type: json
show: results -> {..} -> page -> url & title & tlsValidFrom & tlsValidDays & tlsAgeDays & tlsIssuer

url: https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1
response-type: json
show: results -> {..} -> result & screenshot

Describe the solution you'd like

Allow for comma to be used as you can normally when not working with an array.

url: https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1
response-type: json
show: results -> {..} -> task -> time, results -> {..} -> page -> url & title & tlsValidFrom & tlsValidDays & tlsAgeDays & tlsIssuer, results -> {..} -> result & screenshot

Describe alternatives you've considered

Multiple req blocks, as shown above is the only workaround I can see. In some cases that would be ok, but in others that is not optimal and sometimes it could mean you are getting different results from the api for each request.

Additional context Add any other context or screenshots about the feature request here.

Rooyca commented 3 weeks ago

At the time I tried to make this work but found it impossible, I don't remember exactly what the problem with the commas was. The closest thing to the result you need would be to use req-id and disabled.

The bad: it doesn't work right now (but I think it's an easy fix). The good: the results would be consistent because they would come from the same response. The ugly: you would have to write several code-blocks.

Here is how it would look:

```req
url: https://urlscan.io/api/v1/search/?q=page.url:www.google.com&size=1
req-id: 12
disabled
show: results -> {..} -> task -> time
```

```req
req-id: 12
disabled
show: results -> {..} -> page -> url & title & tlsValidFrom & tlsValidDays & tlsAgeDays & tlsIssuer
```

```req
req-id: 12
show: results -> {..} -> result & screenshot
disabled
```

elvarb commented 2 weeks ago

I think that is a fine solution if the req-id's can be defined dynamically like I mentioned in https://github.com/Rooyca/obsidian-api-request/issues/29

It would allow for flexibility when working with the response. You do one request and then have the option of working with parts of the response where you need it.

For example you could always create the request block at the start of the file, even have a few requests with different req-id's (variable for the filename and some indicator of what the request is).

Then you follow it with a few blocks to pull data from those to add to the file properties if needed.

Then you add blocks for specific data from the request to display along with the rest of the markdown text in the file.

Rooyca commented 2 weeks ago

This plugin is like an amalgamation of code; it has more bugs than features, but for some reason it works. I've been thinking about rewriting it all, but lately I haven't had the time to do it.

elvarb commented 2 weeks ago

I know that feeling well, you start by super gluing code parts to get the original use case working and then it evolves into a monster of both horrors and potential :)

Rooyca / obsidian-api-request

Improve handling of nested JSON responses #34