Help people working exploratively with Aircloak Insights [TLF: 8, 9, 10]

[sebastian]

Report from Vassili:

• He used to work exploratively with Aircloak Insights
• Crucially he did not yet know which queries would yield the best results (right order of GROUP BY clauses, or how many dimensions could be included), so he ran a lot of queries in parallel
• Some of the queries would be good, and others would not be useful, but it would be near impossible to find back to the relevant queries after the fact. The query log would contain both the useful and the less useful queries, and furthermore after a while all the queries look more or less the same, making it hard to tell what is what
• This inability to find the relevant queries makes it hard when you want to find a subset of queries that you want to run again (say make part of an automation), or use in a presentation

Vassili wishes for some way of being able to find back to relevant queries, and some way of adding context so he knows what they are about.

[sebastian]

Please make a proposal and discuss before implementing.

[edongashi]

How about adorning queries with:

comment: string
tags: string[] // in UI as a space or comma-separated string

And the respective UI for them? Then you can view queries of a certain tag and see their comments as well.

[sebastian]

Yes, I was thinking something similar, BUT the UI is going to be the tricky part! How do you make a UX that allows someone to easily find back to old queries in a useful way?

[cristianberneanu]

Adding tags might be overkill for now. We could add a query search page that can look for keywords in the query statement plus filter by date and/or data source. The comment field might be useful if we want to allow query editing after the fact (otherwise, comments could be included in the query statement).

[sebastian]

(otherwise, comments could be included in the query statement).

The primary use case we are trying to solve for here is one where the user doesn't know ahead of time which queries are going to end up useful. I.e. they spray and pray, and later want to be able to mark those that were useful. Because of this adding comments as part of the SQL is not going to be a good approach.

[sebastian]

I think doing either a comment or tags would be enough. Maybe comments is more flexible. Someone can still choose to add #whatever in their comment and then use that to filter later.

[edongashi]

Postgres has some nice text search features AFAIK. We could use that to filter on the comment/tags field.

[sebastian]

Ok, discussed further:

• let's add a comment field to queries. People can use it as they please. They can add an . or an x to simulate just marking a query. They can add #my-best-query to tag. They can add freeform text to add more context
• we provide a query overview page showing a user all their queries
  • Make it possible to only show queries with comments
  • Make it possible to filter by a search term. Like @edongashi writes above, Postgres has excellent text search. Let's make both the data source name, the comment and the query statement part of what is searched
  • When clicking on a query, you get the equivalent view that you would get if sharing a query. I.e. a detail view of the query and the query result
  • It should be possible to delete a comment from the overview page (akin to untagging a query), but we don't need to support changing a comment there, unless it ends up being trivial...)

[sebastian]

Note: I envision the query page to be across all queries for a user across all data sources! I.e. it could be a button next to the data sources button in the top left:

[edongashi]

Some questions:

• Do we show in-progress queries in this page? This may complicate implementation.
• Do we show results if the query was successful? This may cause bloat.
• Can we filter by date? This allows to jump to the past without clicking "Load more" so often.
• Can we filter by data source like in admin's auditlog/queries? I think this is better than including the data source in the search term.
• Can we delete queries from this page? Can we set notes on queries from this page?
• Should we go with a live view or a react page with API? Both have their pros and cons, and also depends on the answer of the previous question.

[sebastian]

Do we show in-progress queries in this page? This may complicate implementation.

I would say yes, but in no different way than we do for queries that are completed. Basically just a complete list of the queries that originate from a user irrespective of their state.

Do we show results if the query was successful? This may cause bloat.

I'd say no. You get the answer by following the link to the query page.

Can we filter by date? This allows to jump to the past without clicking "Load more" so often.

It would be nice. Like the time filter we use quite often in the admin section.

Can we filter by data source like in admin's auditlog/queries? I think this is better than including the data source in the search term.

Yes, you are right. It would be better. If we have a filter side-bar anyway, maybe we can add filtering by data source there.

Can we delete queries from this page? Can we set notes on queries from this page?

Delete queries: no. See notes: yes.

Should we go with a live view or a react page with API? Both have their pros and cons, and also depends on the answer of the previous question.

I would personally prefer a live view, but I'm happy to let you take a lead on this as you go about implementing it.

[edongashi]

Thanks for the detailed response!

See notes: yes.

What about setting notes, i.e. will the query be entirely immutable? I'm asking because the mutation side is in react and makes the decision more difficult.

[sebastian]

Let's make the notes immutable on the query list page. If you want to edit or delete the note you can click into the query view. I think we are offering enough functionality this way, and reduce the complexity of the implementation significantly this way too.

[edongashi]

If you want to edit or delete the note you can click into the query view.

Unfortunately, the permalink is immutable too, and we don't provide a way to navigate to a specific query from the data sources page. This means that mutating (set note or delete) an old query would be very difficult because you have to find it by scrolling in the data source.

[sebastian]

Oh, I see! Thank you for clarifying! In that case I think we might have to find a way to edit the notes after all. Which do you think is easier? Allow note editing on permalinked queries in the case of the viewer being the query runner, or allowing for inline note editing in the overview?

[edongashi]

I think allowing users to set note from the permalink is easier since it's in react already. Making the history view immutable will save us some complexity, which seems to be high enough already...