Currently, the endpoints for marquez all look along the lines of:
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}
This means that namespaces are effectively mandatory when users are working with the api and thus the web contains a dropdown allowing users to select a namespace. This is not ideal for a few reasons:
We either need to default a namespace (1st in the list) or force a user to select a namespace before anything else can be rendered.
A user does not always know the namespace of a job or dataset they are looking for because many of the OL integrations (Spark, Airflow, etc...) assign these.
A long list of namespaces is not going to be ideal since a MQ user with thousands of namespaces will need to scroll a long way and have application rendering degraded.
Therefore, we need to handle namespaces a little differently across the board to rectify these problems listed above.
Proposal
Migrate the format of the endpoints to use query parameters like below and then forward old formats to new formats for a few releases (or forever)
# OLD
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}
# NEW
http://localhost:5000/api/v1/datasets/{dataset}?namespace={namespace}
Update the web to just render all the jobs and datasets, paginated of course without regard to namespace. Include a namespace filtering mechanism with a typeahead select so that users with many namespaces can find there known namespaces and filter.
Currently, the endpoints for marquez all look along the lines of:
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}
This means that namespaces are effectively mandatory when users are working with the api and thus the web contains a dropdown allowing users to select a namespace. This is not ideal for a few reasons:
namespace
of a job or dataset they are looking for because many of the OL integrations (Spark, Airflow, etc...) assign these.Therefore, we need to handle namespaces a little differently across the board to rectify these problems listed above.
Proposal
Migrate the format of the endpoints to use query parameters like below and then forward old formats to new formats for a few releases (or forever)
Update the web to just render all the jobs and datasets, paginated of course without regard to namespace. Include a namespace filtering mechanism with a typeahead select so that users with many namespaces can find there known namespaces and filter.