e2b-dev / E2B

Secure open source cloud runtime for AI apps & AI agents
https://e2b.dev/docs
Apache License 2.0
7.02k stars 458 forks source link

Improve file watcher in Python Sync SDK #457

Closed jakubno closed 1 month ago

jakubno commented 1 month ago

Description

Current solution didn't work well with in sync Python. If you're unsure whether a file has been created, you might end up waiting for a timeout to exit the loop.

New implementation uses polling, you can ask for events whenever you want and it will request envd to send all events (or only new ones). This should make it much better for users with sync Python SDK

Example of a problematic usage before and after:

Before

sbx = Sandbox()
watcher = sbx.files.watch("/home/user")
sbx.files.make_dir("test")
for event in watcher:
    print(event)
     # !!! if you don't exit, you would be stuck for the rest of the timeout (default 60 seconds)
    break
watcher.close()

After

sbx = Sandbox()
watcher = sbx.files.watch("/home/user")
sbx.files.make_dir("test")
events = watcher.get_new_events()
watcher.stop()
for event in events:
    print(event)

Even worse case was if you don't know if anything will happen:

Before

sbx = Sandbox()
watcher = sbx.files.watch("/home/user")
if random.random() > 0.5:
    sbx.files.make_dir("test")
# There's 50% chance you will get stuck. The only workaround is to run in in separate thread and you kill it after a while (you aren't really sure when it's safe)
for event in watcher:
    print(event)
    #  !!! if you don't exit, you would be stuck for the rest of the timeout (default 60 seconds)
    break
watcher.close()

After

sbx = Sandbox()
watcher = sbx.files.watch("/home/user")
if random.random() > 0.5:
    sbx.files.make_dir("test")
events = watcher.get_new_events()
watcher.stop()
for event in events:
    print(event)
changeset-bot[bot] commented 1 month ago

⚠️ No Changeset found

Latest commit: bfc4bb51d8cddcf9ecb709c5b7118ada1f47e206

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ValentaTomas commented 1 month ago

Few notes:

  1. Do we agree on the polling design?
  2. Protobuf field should be snake case
  3. I'm thinking about how this interacts with the reconnect — the client watcher keeps around the offset + watcher id, so when you reconnect you effectively cannot reconnect to the watcher. Could be ok, just pointing this out
mlejva commented 1 month ago

A few notes to both SDKs: