Initial File System

[colinjneville]

• [x] File Permissions
  • r/o and r/w files are kept in separate directories with the appropriate preopen permissions.
• [ ] Live Update
  • [x] Implementation
  • [ ] Testing
• [x] Golem manifest
  • [x] Initial File System contents parsing

    files:
    - sourcePath: "~/my_files/readonly"
    targetPath: "/ro/"
    permission: readOnly

  • [x] File packaging (local and web)
  • Deflate (can be changed easily if needed) zip files
  • [x] REST integration
  • [x] Update golem-openapi-client-generator to support requests with either multipart or non-multipart definitions
  • [ ] Full manifest usage(?)
  • The implementation takes a --manifest argument for create/update and only uses the files section.
• [x] REST API
  • [x] Component API
  • [x] Create
    • Read-only and read-write files are packaged separately as multipart properties files_ro and files_rw. No separate permission parameter is needed.
  • [x] Update
  • [x] Worker API
  • [ ] API Definition API
  • [x] Theoretically implemented
  • [ ] Testing
  • [ ] Spawn worker if no name is provided
• [x] Worker Gateway
  • [x] file-server binding type
  • [x] Shorthand Rib result handling
  • [x] mime type inference
• [x] Component Service
  • [x] Store file archives in blob storage alongside component binaries
  • [x] Cache files
  • r/o cache is an Option<Arc<TempDir>> of a folder with the extracted files
  • r/w cache is an in-memory archive
  • Current cache is not ideal, since archive size is not taken into account, and a component with no files still takes a spot in the cache
• [x] Worker Executor
  • [x] Read filesystem implementation
  • [x] gRPC API
  • [x] Preopen

For my implementation I packaged the read-only and read-write files separately, and store them both as archives in blob store alongside the component wasm. The component service caches a TempDir handle to a folder with the extracted read-only files to be used by all instances. The read-write files are extracted to the worker instance temp dir during creation.

This change also relies on a few changes to golem-openapi-client-generator to support mixed response types, etc. needed for backwards-compatible CreateComponent/UpdateComponent. Let me know if I should open a pull request on that repo.

Mounting multiple preopens to the same path does not seem to work properly (it seems like only the first applicable preopen is tried), so read-only files use a dedicated path (/static/) for now. Note that paths with the r/o directly work as expected, but it will not show up when doing a directory listing on /. Maybe in the future automatically create a symlink for each read-only file, or at least one to the r/o folder?

/claim #1004

[algora-pbc[bot]]

💵 To receive payouts, sign up on Algora, link your Github account and connect with Stripe.

[jdegoes]

@colinjneville Congratulations on submitting your solution for our Bounty-to-Hire challenge (#1004)!

We have reviewed your pull request and believe that it is very promising. While not yet satisfying all of the requirements of #1004, we believe there is a short and well-defined path to get there, and so we are delighted to announce that you are a finalist in our Bounty-to-Hire event!

Next steps:

1. You are guaranteed an interview based on your work thus far.
2. Should you continue to work on this pull request, and get it into a state that can be merged to satisfy all requirements, you stand a reasonable chance of winning the bounty.

If you no longer wish to work on this pull request, please close it, and we will reach out about the interview. On the other hand, if you wish to continue pushing this pull request forward, then please let us know here and continue your work.

Congratulations again on being a finalist for Bounty-to-Hire!

[jdegoes]

@colinjneville

We have reached November 4th, which is the day we must announce the winner of the Bounty-to-Hire program!

Given the scope of the challenge, it's incredibly impressive that we have had not one, but THREE finalists, who each demonstrated high competence and skill in tackling a problem that would stump most engineers on the planet.

However, ultimately, there can be only ONE WINNER to the Bounty-to-Hire challenge. So after MUCH analysis by Golem engineers, thoughtful consideration, and spirited discussion, we have decided to pick a winner.

That winner will be announced on our LIVE STREAM event TODAY at 12 NOON ET:

https://www.youtube.com/watch?v=at8EPqLWIRE

Without question, all three finalists are highly qualified Rust engineers and we look forward to interviewing and getting to each of them, with the intention of making at least one job offer to fill our recently opened position.

While you await the announcement of the WINNER, please shoot an email to john@ziverge.com to schedule your interview!

Stay tuned for more.

[jdegoes]

@colinjneville If you plan to join for the Live stream today, please do send me a message on Discord (you can find me on the Golem Discord), or an email at john@ziverge.com. Thank you!

[jdegoes]

@colinjneville Thanks for your work on this feature! We remain excited to interview you, should you be interested in doing this sort of work full time as part of the Golem Cloud engineering team.

Since announcing that Maxim Schuwalow has won this bounty, we are now closing this pull request, and look forward to posting many more bounties on smaller-sized issues in the near future.