Providing the recording as a URL query param instead of a file

[Zac-HD]

I'd love to be able to use pytrace to debug failures from CI... but getting files off CI and then onto my local machine before I can view them is a pain.

Would it be possible to pass this data as a URL query param instead? If so, dumping that URL in the logs would make time-travelling debugging as easy as clicking a link 🤩

[gleb-sevruk]

Are we talking about WebUI ability to open remote recording?

I guess it is already implemented, try https://app.pytrace.com/?open=https%3A%2F%2Fpytrace-demo.s3.eu-central-1.amazonaws.com%2Fsession.chunked

The few culprits are:

• you need to remove ending .pycrunch-trace, so https://pytrace-demo.s3.eu-central-1.amazonaws.com/session.chunked.pycrunch-trace becomes https://pytrace-demo.s3.eu-central-1.amazonaws.com/session.chunked
• There is no loading indicator implemented for remote files (and it is one of the reasons this feature is hidden)

Code for reference: https://github.com/gleb-sevruk/pycrunch-tracing-webui/blob/master/src/store/index.js#L184

[Zac-HD]

I was thinking "just click https://app.pytrace.com/?data=base64_encoded_payload_here "; i.e. putting the whole recording (compressed and encoded) in the URL. No data storage at all!

A similar idea would be to load the recording from a data-URL, like data:application/octet-stream;base64,SGVsbG8sIFdvcmxkIQ==. Same goal - make it easy to use pytrace recordings from systems where you only get text logs - though a somewhat more complicated user experience.

[gleb-sevruk]

Ok, I ran a few checks. Previously I was thinking that url length limited to 32k bytes; but now Chrome states it can support up to 2MB url string (https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_display_guidelines/url_display_guidelines.md#URL-Length)

I tried compressing relatively small recording of 36MB, and it won't fit into limit:

As I agree this feature may be useful, it will not work under all circumstances. (what should recorder do, if predicted compressed size will exceed 2MB?)

Maybe it worth looking into alternatives, such as S3-compatible storage (Ceph, Minio, AWS) or just exposing artifacts from CI

[Zac-HD]

My thoughts on this are basically "can we make the data smaller", then "can we compress it better" (e.g. sorting it, canonicalising, using zstandard instead of zip, etc), and if that's too hard just tell me that the feature I want isn't practical!

(in any case, my sincere thanks for an awesome open-source tool and your quick responses here :smiling_face_with_three_hearts:)

[Zac-HD]

The obvious alternative is local recordings, so I experimented with that for a bit. My goal is to have a link which I can print to the terminal from pytest, and the user can click to load directly into the debugger for that test. (asking the user to load it is pretty nasty, since there might be hundreds of recordings)

1. I tried passing local filepaths as the ?open= param, but promptly ran into an XSS issue: xhr.js:178 Not allowed to load local resource: file:///C:/Users/Zac/Documents/GitHub/hypothesis/pycrunch-recordings/run/session.chunked.pycrunch-trace. OK, fair enough, allowing JS to open local files would be a horrendous security problem for Chrome.
2. Next idea, spin up python -m http.server to serve the files over http. No dice: ... No 'Access-Control-Allow-Origin' header is present on the requested resource. Also fair, and this one I can work around by fixing the local server.

[gleb-sevruk]

python -m http.server I was thinking this too in order for integrate PyCrunch test run with time travel debugger 😀

But the obvious problem is disk space

[Zac-HD]

Well, the trace has to be stored somewhere, so uploading it just moves the problem around (and is impractical for people on slow internet connections). Some combination of compressing traces and aggressively deleting old trace files would probably be sufficient.