flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

Adds a dedicated Python API #27

Closed ilumsden closed 10 months ago

ilumsden commented 1 year ago

This PR adds a dedicated Python API to DYAD. This API will not be built/installed by Autotools. Instead, it will use setuptools/pip for its build/install process. This will allow us to host the Python API on PyPI if desired.

In terms of the interface, there are only 3 things a user may have to interact with:

  1. The dyad_open function
  2. The dyad_open_local function
  3. The Dyad class

Most users will only use dyad_open. This function is a drop-in replacement for the built-in open function. Internally, this function is a "context manager" (see this page for more info). It creates a global Dyad object (if not created previously) and uses the Dyad object and built-in open function to carry out data production/consumption. When using dyad_open, DYAD will always be configured with environment variables.

The dyad_open_local function is provided to allow users to programmatically configure DYAD. When using dyad_open_local, users will first manually create a Dyad object and initialize it with the init or init_env methods. Then, users will invoke dyad_open_local. This function takes the same arguments as the built-in open plus an additional positional argument called dyad_ctx. This argument is the last required positional (i.e., it comes after all positionals from the built-in open function), and it accepts a pre-initialized Dyad object.

This PR is currently work-in-progress. The API is mostly complete, but packaging (i.e., setuptools stuff) and basic testing are still required.

JaeseungYeom commented 1 year ago

Is this PR is off the unmerged DTL PR? The reason I am asking is it is difficult to see what changes are relevant to the purpose of the PR.

JaeseungYeom commented 11 months ago

Is is still work in progress PR?

ilumsden commented 11 months ago

All that's left is testing to make sure it works correctly. Haven't gotten to that yet.

ilumsden commented 10 months ago

@JaeseungYeom @hariharan-devarajan @vsoch now that the DTL PR is merged, I've fixed up this PR. It's now fully ready for review.