Open JPvRiel opened 2 years ago
you are trying to use Kqlmagic objects in a way they were not meant to. Kqlmagic main purpose is to query data stored in Kusto/ADX, Application Insights, and Log Analytics using the KQL query language within a notebook. The intergration with Python is limited to the result tables as dictionary or dataframes, plus some extras. raw_json was not designed to be used for programming. Its purpose is only to show the raw result.
I will consider to refactor those properties in next version, in a way, that they can be used programmatically by python code.
@mbnshtck thanks for the reply.
Fully appreciate that it's abnormal to need to to dig into raw_json
and typical usage will be using a pandas dataframe, etc, so this won't affect many users.
The pretty formatting object and lexer feature overlaid on the JSON is nice when someone want's to view a small result/API response. Just noting that the name threw me off and it's impractical for debugging exactly what happened with a large JSON API response.
_json_response
in the end was good enough for me to debug what was going wrong (e.g. how I found the detail of why my dateframe was not fully populated due to API limits as per https://github.com/microsoft/jupyter-Kqlmagic/issues/77).
Your call if you want to risk maybe changing var names to be a bit more intuitive (could be breaking change for some notebooks), although people probably don't commonly use these for much other than debugging...
Since it would be for advanced debug use, it could also maybe be somewhat hidden as a private var I suppose? E.g.:
_raw_json
or _raw_json_response
can be the unformulated pure JSON. Then raw_json
and _json_response
remains as they are (no breaking change).
Once you settle on which bits make good handles to debug with, I'd be happy to contribute a PR on docs on how to debug a bit given the raw response versus what ends up in the dataframe.
If a user wants to inspect and work with the raw JSON response while debugging,
_kql_raw_result_.raw_json
does not provide a a raw JSON string.Instead, it's a
FormattedJsonDict
object.Perhaps
_kql_raw_result_.raw_json
should be the actual raw JSON? And a new attribute_kql_raw_result_.formatted_json
would then be a better name for what is actually given.E.g. I was attempting to understand if Azure log analytics response limits return something in the up to ~61MB JSON response, and it's not practical to view
_kql_raw_result_.raw_json
at this size, and it breaks the option to use something such as jmespath.This can't be done.
Nor can this:
As per:
There are two JSON related attributes, but neither are proper JSON that can be debugged as JSON using standard python JSON libs. They're all already preprocessed.
['_json_response', 'raw_json']
Even
_json_response
is poorly named given it's actually a python dict, not a JSON string. If it was name_response
, it would not invoke the assumption that the type is a JSON string.