search

plynx-team / plynx

PLynx is a domain agnostic platform for managing reproducible experiments and data-oriented workflows.
https://plynx.com
Apache License 2.0
301 stars 37 forks source link

Read numpy input, preview code, test_run/quick_run #88

Open thoth291 opened 3 years ago

thoth291 commented 3 years ago

  1. I've created 2 operations. One creates a numpy array: data_output = np.arange(15).reshape([3,5]) And another one adds 10 to that array: output_data = input_data + 10 I don't understand how to do that - and can't find examples to learn from. Can someone help me with that or point to better docs?

  2. How to preview an operation's code from within the workflow graph? Does it always using un-depreciated version or one can still point to depreciated one? Id's are non-human readable - so navigating in operations tab might get more difficult with many similarly named operations.

2.5. How to completely delete an operation or perform a cleanup by removing unused ones into some sort of archive?

  1. How to test an operation without moving to a workflows and creating a test workflow? It would be great to create a quick run option before approving an operation for the final workflow. Technically speaking I could of created a simple workflow to test my currently-in-developement-not-yet-approved operation, and once it's tested I could approve it and push to the actual workflow where it could be used.
khaxis commented 3 years ago
  1. In the current state operations use files to share data between each other. In your case you will need to write your np array to a file and read it from another operation 
    with open(outputs['output'], 'w') as f:
np.save(f, data_output)
....
# Operation 2
with open(inputs['foo']) as f:
input_data  = np.load(f)
output_data = input_data + 10

Communication using the files is a use basic case. This way is easier to distribute computation across cluster of workers, also using operations written in different languages. But it does not have to be this way. Plynx supports Executor plugin, that can interpret code and graph itself in your way. Please let me know if you want to hear more about it! Here are some basics https://plynx.readthedocs.io/en/latest/plugins.html#executors

khaxis commented 3 years ago
  1. Click on an operation. On the right panel, a link to it will appear under "this operation". Currently the only way to preview it. You think it will be useful to improve it? 2.5. I am afraid you can not delete it. It's actually rather legacy from previous version and one use case. Without going too much into details, the operations are kind of deleted if you deprecate them with "MANDATORY_DEPRECATED" flag. That means users cannot reuse them in any of the graphs. Would it be helpful?

Actually not the first time people are asking about removing an operation! Thanks for noting, because I need more information about use cases

khaxis commented 3 years ago
  1. I am pretty sure, that would be an amazing feature! Unfortunately, didn't have time to do it. There is a lot of things to think of and implement. You think it should be higher in priority?

Currently you can click "preview" and run the resulting script locally... Not best option but definitely easiest for now