microsoft / TaskWeaver

A code-first agent framework for seamlessly planning and executing data analytics tasks.
https://microsoft.github.io/TaskWeaver/
MIT License
5.38k stars 689 forks source link

How to use Taskweaver online? re; project directory #233

Open antoineross opened 8 months ago

antoineross commented 8 months ago

I was wondering if it's possible to access the a cloud bucket storage instead of a local project directory is that possible?

Or is there another way to handle this?

Thank you in advance!

liqul commented 8 months ago

Currently, we only support using the local file system. If the cloud storage could be mounted to the local file system, it would be no problem. Otherwise, we need to re-implement all the file i/o operations w.r.t a cloud storage, which need quite a lot of efforts.

mirojs commented 8 months ago

I believe this is exactly the reason why Taskweaver developers would expect MS could help to implement this. If they do it by themselves, it may take even longer time and each individual developer has to repeat the same thing. At the end of day, it's the end customer who would require the cloud storage and Taskweaver would be able to reach even more customers with this necessary feature. Hopefully this helps.

liqul commented 8 months ago

I believe this is exactly the reason why Taskweaver developers would expect MS could help to implement this. If they do it by themselves, it may take even longer time and each individual developer has to repeat the same thing. At the end of day, it's the end customer who would require the cloud storage and Taskweaver would be able to reach even more customers with this necessary feature. Hopefully this helps.

You are right that we are trying to build the infrastructure to ease the development of agents. We shall put this in our backlog and would like to see more support on this. Given our resource constraint, we need to be very careful on what are we working on.

mirojs commented 8 months ago

Thanks.

antoineross commented 8 months ago

Hi I think I have a solution. Currently I am using a temp directory for the project directory, I just now need to make sure that every artifact generated or accessed by the LLM is through that temp directory.

I noticed that TaskWeaver stores file in quite specific areas. May I know where I can set these in the source code? I just need to be able to point them to the temp directory:

The values of variables of the above Python code after execution are:

time_series_map_20240306.png

The following artifacts were generated:
- type: image ; uri: file:///private/var/folders/jf/d3lxj_s12bv6xfgmxqxsjnd00000gn/T/tmpkfoo5mat/workspace/sessions/20240313-185034-99ad1708/cwd/post-20240313-185324-e8b62c34-display-1_image.png ; description: <Figure size 2400x1200 with 1 Axes>
antoineross commented 8 months ago

To anyone who wants to reproduce just use the Web UI and clone the project directory into a temp directory then pass that into the

app = TaskWeaverApp(app_dir=temp_project_path, use_local_uri=True)

liqul commented 8 months ago

To anyone who wants to reproduce just use the Web UI and clone the project directory into a temp directory then pass that into the

app = TaskWeaverApp(app_dir=temp_project_path, use_local_uri=True)

You can place the project folder at any place assessable from the local file system.