Fix/enable-handle-fs-connector

[kirtimanmishrazipstack]

What

• Azure FS, GCS have been enabled
• Made necessary changes to comprehend all unstract fs connector
• Remove path in minio, not used
• Remove overwrite parameter from destination connector

Why

• Remove path in minio, because path is not used. In test connection, we are only using bucket.
• Remove overwrite parameter from destination connector. This is done because all fsspec connector is not using overwrite parms. Connectors like Dropbox, minio, GCS, Azure FS will always overwrite the file if present. Where as Gdrive always creates a new file if already present. So this param is not needed in destination connector

How

• Azure FS, GCS have been enabled from FE
• Made necessary changes in backend/workflow_manager/endpoint/source.py and backend/workflow_manager/endpoint/destination.py to comprehend all unstract fs connector
• Remove path in minio from unstract/connectors/filesystems/minio/minio.py
• Remove overwrite param backend/workflow_manager/endpoint/static/dest/file.json to remove overwrite from destination connector

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• This PR only affects the FS connector. All FS connector have tested on local

Database Migrations

• N/A

Env Config

• N/A

Relevant Docs

-

Related Issues or PRs

-

Dependencies Versions

-

Notes on Testing

• Tested all 5 FS connectors as a source and destination on workflow

Screenshots

Checklist

I have read and understood the [Contribution Guidelines]().

[kirtimanmishrazipstack]

Changes look good for the most part, however do confirm if this requires any migration to avoid regressions on removing the parameters. I believe we persist these details in the DB

@chandrasekharan-zipstack I believe we are storing as json in workflow_endpoints table. When I remove the params, its will only change the json. So no migration changes as per my understanding. @muhammad-ali-e let me know your thoughts

[muhammad-ali-e]

@chandrasekharan-zipstack I believe we are storing as json in workflow_endpoints table. When I remove the params, its will only change the json. So no migration changes as per my understanding. @muhammad-ali-e let me know your thoughts

Yes I beleive so.It will not affect anything if it not used anywhere.

[github-actions[bot]]

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

[sonarcloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud