Share tab data via link

[lukashornych]

When several developers (usually WD with AD) are debugging a problem with data or query, they need to share query with variables. But sharing these information using messaging application is tedious and may lead to misunderstanding and not 1:1 copy of the original query/variables.

We could leverage the demo query opening feature with the future planned functionality of storing opened tabs in local storage for session restoration to allow providing a URL link that would contain actual URL of the evitaLab and base64-encoded object with opened tab (in which the link was created) with all the data in the tab (query, variables, ...). This way one developer could easily share entire debugging stack to other developer just via link.

One problem is that we didn't want to allow custom queries to be passed as URL parameter other than safe ones from documentation. We fear that someone may misuse the functionality for attacks, so this will need to be discussed further.

cc @novoj

[novoj]

Well, we could allow to pass / accept them, we cannot do that automatically and immediately execute for sure. For me it would be ok if query from non-trusted source is accepted first with modal dialog with text: "The URL contains a request from a non-trusted source. Do you want to accept it and add it to your query editor (you still need to execute it manually afterwards)?" and if you click okey, the query will only be pre-filled in the editor, but still as "passive" text until the developer manually sends it to the evitaDB server.

[novoj]

And by the way, the URL has safe limits of about 2-4k characters - what if the query doesn't fit in there? We can compress it somehow - but we should take it into consideration.

[novoj]

@lukashornych when it's done - let's make from this a catchy video to share on social links. Script:

1. Screen-capture from person one - writes query with variables
   • generates shareable link
   • goes to discords and sends it to another person
2. screen rotation effect to make obvious we change seats
3. Screen-capture from person two
   • receives the share link
   • clicks on it
   • confirms the modal dialog - the query is in the editor
   • executes the query - and we have the result
4. some effect at the end

I can take care of the post-processing (or my daughter can :) ) - so let's just record the source videos. I think it could have a like or two ;)

[lukashornych]

@novoj Great idea with the video :+1:

[lukashornych]

Been experimenting with the sharing format of data...

I've used a larger query which seems to be pretty standard size of queries in FE of our e-shop application:

{
    queryProduct(
        filterBy: {
            attributeCodeInSet: []
            entityLocaleEquals: cs
            priceInCurrency: CZK
            priceValidInNow: true
            priceInPriceLists: ["basic"]
            attributeProductTypeInSet: ["BASIC", "SET", "MASTER"]
        }
        orderBy: {attributeCodeSetExact: []}
        require: { priceType: WITH_TAX }
    ) {
        recordPage(number: 1, size: 20) {
            first
            empty
            pageNumber
            totalRecordCount
            pageSize
            data {
                primaryKey
                priceForSale(validNow: true, currency: CZK) {
                    priceWithTax
                    priceWithoutTax
                    currency
                    validity
                }
                price(priceList: "reference", currency: CZK) {
                    priceWithTax
                    priceWithoutTax
                    currency
                    validity
                }
                stocks {
                    attributes {
                        quantityOnStock
                    }
                }
                attributes(locale: cs) {
                    availability
                }
                attributes(locale: cs) {
                    name
                    descriptionShort
                    url
                    productType
                    ratingVotes
                    rating
                    minOrderQuantity
                }
                master {
                    referencedPrimaryKey
                }
                tags {
                    referencedEntity {
                        primaryKey
                        attributes {
                            name
                            code
                            isVisibleInDetail
                            isVisibleInFilter
                            isVisibleInListing
                        }
                        type
                    }
                }
                parameterValues(filterBy: { attributeVariantEquals: true }) {
                    referencedEntity {
                        primaryKey
                        attributes {
                            name
                            code
                            order
                        }
                        parameter {
                            attributes {
                                orderInParameter
                            }
                            referencedEntity {
                                attributes {
                                    code
                                    isVisibleInDetail
                                }
                            }
                        }
                    }
                }
            }

        }
    }
}

and variables

{
"variables": {
    "filterBy": {
      "attributeCodeInSet": [],
      "entityLocaleEquals": "cs",
      "priceInCurrency": "CZK",
      "priceValidInNow": true,
      "priceInPriceLists": [
        "basic"
      ],
      "attributeProductTypeInSet": [
        "BASIC",
        "SET",
        "MASTER"
      ]
    },
    "orderBy": {
      "attributeCodeSetExact": []
    },
    "require": {
      "priceType": "WITH_TAX"
    }
  }
}

which results in approximately 3500 characters when a transfer object is constructed. I've tried lz-string JS library to compress the raw string with query and variables into URL safe string

N4IgjgrgpgTgniAXCYAdAdgAm5ys4AKMA9gCYQDGALgBQY4OYBmAlgDZWwBCcimaWRkICGVKjBYAjCJwDCZKAEl0AZShU+AbQC69IYyjoqLKnAAyxCsLZQAopGsBnPhUd79OAA4SKS9LIgYGEMKXkxZAC0AaXcPTG8WXwA1axZSZQA5YgB3PnFoWI8E32UiRKgzFkcqZ0xNVBBJYUdEht1BOMxRcSkZKCIySioAFThPPzUNOoauAEEVRVkGgBpMBpVbYZW1kABZeeHbACU2wpwAXzPsYhhSbjDgbolpOQVJ2wAPYWotbUuOxjBSAsYJ8YDxHxQUbjPgAdUUwwAEgB9YazAAamH+DAAlPwrphghQbqQCMIAOZQGjoCAAW0ksD4AEZVi0AF5QPgAJgADHiBJ1sKwYNUCQwoLTPKYxV4KVAMnSGTAZdgqMQqNYjlBibd5BAjCr4nKVCwOYbSKJhPiAYKISxacJ4FEoHBDQxilAAGI3FTWKkAN1SpCyuUw+SgqwogWC6FCfEiUX5bv0HthJgAFsNhB9k0JUxniDIszmbbacFGgiFXaWy5hA2w0iZq7XsNiWx6aB7KtU+A1gkxYCEoNsKzG4+Fokma2X81RM9nc4xZ+nCyMF9PbaOq4uGPXG9KN-o27XqpYANaOa0txhPXqcS8C6-6BxGJsAeVUaooZ53F0Xx7LW8XigRwaDYSw-RcRwpyfHBhEDdhhEkdgm3-RcgL6UDwKsGwoJg2DMHQYRaWHQ9OjuRwKAkKUWGIVQVxgKhf2wQI2GYiFBmoaFSIIwlRBYdBySSdUQPYmB+ME9jaQEt9blgABFCBhFfA9rwA20HWqWArwI-tB1jKBSQkB0nRdNCyMYDVyQfMSoAHMdDNsFS4B03jsASEy4GdZs3OwDD71c3ycCIkj2I8Yk7jC-QqiSKopBsZQABF1GEdgoqEGK4skBL0E9dhOGVCzr0ylpsr8btjEkorBXUgjTHGdjas6Jq4k8R1iPUWAUjYaBQNYDh7jBLoxGePoUgkZSqHsJS2FqcMsXw2C9Ic0gnOMUxAt4jzHS8szqs6fyQM2oKQp4oKGAis7zuuOTCvOlrrza8SSIK47fMOmz9qfElYFKdqXtgdLGAe3S7P03xVuct7ro+6HrvLBQgdtEr4r8ZKNTSr61KRv8saPRrzNtf4MAJAD-nOEBlhAQMJrKxwkBQPQGhplgkJsNwkGtVAmJAfqCp4BowRJqhsAaQ75DuZRJkFuptGWYXRZAQx1vMCCbGmpwZYaVwVgVnYPWUAJK1jV1OYaBNdfQbnFY9bq0kyHIZfDeWrZF-XIVKSEKo5rRhe5homhaCgGj1uW9bFka736EhyC4sYJnUGX6ld-2QDmBYlkpv2eY2LYs5Tnn9hUQ4ThAUPhfOF3U5+mABc5gRrZ2cW3nUT5vh534K6rnmgQgEFh3r8OQA9bitZAeEkVRDEQ9dttyZAc4gA

which results in approximately 1500 characters.

Which is not too far from the URL limit (2048 characters), but personally, I think this should work for the majority of cases. For the rest of cases where the payload is bigger that the URL limit, we could either not allow to create a link or we could copy only the encoded data string into the clipboard and then create an input in the lab to allow pasting the string into. This would not be as pretty as the link, however it could still be better than sharing raw queries.

There is another problem I've came across, is that the shared object would contain reference to used evitaDB connection. The problem is that usually, connections don't have fixed ID. Instead evitaLab generated random UUIDs for the connections. We do have support for custom IDs in evitaDB config file and it is used for demo dataset, but it is not required to fill the ID. Altough we could require the custom static ID for the preconfigured connections in the evitaDB config. Another problem could be user-created connections (which I doubt would be used much for this sharing feature) because these are stored only in a browser and there is no way we can reliably match these IDs across multiple browsers.

What I propose is this:

• all preconfigured connections will need to have static IDs
• if a tab is to be shared in preconfigured connection (shared for all developers) AND the final data object string fits into a URL, a link with domain, evitaDB connection, query etc is copied to the clipboard MAJORITY CASES
• if a tab is to be shared in preconfigured connection (shared for all developer AND the final data object string DOESN'T fit in a URL, only the data object is copied into the clipboard BUT with a preselected connection
  • target user will then have functionality to paste the string into an evitaLab instance
• if a tab is to be shared in custom connection, the final data object string is copied into the clipboard WITHOUT preselected connection
  • target user will then have a functionality to paste the string into an evitaLab instance AND will have to select a connection from a select widget

Realistically we could simplify it only to the first use-case for now as I think this will cover all cases we need right now. The use-cases would simply not be possible yet.

[novoj]

I agree with your conclusions. Perhaps I have a follow-up idea. We don't generate the connection ID entirely automatically - the base of the ID is always a catalog name followed by a - character and a hash of the catalog location and a few other things. If we split the ID and keep only the part before the - sign, matching might be a lot easier. Don't you think?

[novoj]

And chatGPT thinks (https://chat.openai.com/share/527ccaa0-098d-4c65-a5ae-ce5a74967772) that the URL length limit is much higher than 2048. So maybe we should always try to generate the link. What about leaving the option to paste the base64 encoded text for the future if we encounter a frequent problems?

And this post confirms it: https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers ... again only M$ sucks.

[lukashornych]

I think you mixed connections and catalogs together. Connection is just a pointer to a evitaDB server, it doesn't have a single catalog associated with it. But maybe we could use the evitaDB server URLs, normalize them and create hash as ID from it. But we would also have to validate duplicate URLs across multiple connections, which may be problematic in that that a connection can have multiple URLs for different APIs, however, the base Lab API URL is always required, so we could just check and hash that.

Regarding the URL length, ChatGPT doesn't work for me now, but I've found another helpful article where it says the same thing I found originally. The Undertow server used by us to server evitaLab have higher limit apparently, I tested 5500 characters and it didn't complain.

Maybe we could show a warning if its longer that it may not work in some browsers? And like you said, the pasting option can be implemented in future if needed.

[lukashornych]

Everything is implemented. Keeping it open to record the video after it is fully released and deployed on evitaDB demo.