search

ipfs / ipfs-webui

A frontend for an IPFS Kubo and IPFS Desktop
https://webui.ipfs.io
MIT License
1.57k stars 490 forks source link

feat: bulk CIDs import #2173

Open asigalov61 opened 1 year ago

asigalov61 commented 1 year ago

Hey guys,

I was wondering if it would be possible to add a bulk files/CIDs import from a txt or csv file?

I have a list of CIDs but there is no option in IPFS-Desktop to import them in bulk/automatically.

Can you implement this option? Or, if it is possible to do via Kubo or some other script method, it would be nice to see such method in the docs.

Please let me know.

Alex

welcome[bot] commented 1 year ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

SgtPooki commented 1 year ago

@asigalov61 Great question. You should be able to do this very easily by dragging and dropping files onto the IPFS-Desktop "Files" screen. Please let us know if you experience any issues with this.

asigalov61 commented 1 year ago

@SgtPooki Thank you for your response. But I was asking about download, not upload.

I have a csv file with IPFS CIDs and I want to import all CIDs so that I can download them.

There seems to be only option to do it one by one by going to Files -> Import from IPFS, but there is no bulk "Import from IPFS" option.

Am I missing something?

Please let me know.

Thank you.

github-actions[bot] commented 1 year ago

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

SgtPooki commented 1 year ago

@asigalov61 I see! sorry for my confusion.. I think an option to bulk import CIDs would be a great addition to IPFS-Desktop. Just FYI: that change would likely live in the ipfs-webui repo.

It might be a while until we get to this feature since a lot of our focus has been on Helia, but to unblock yourself for now, I threw together https://github.com/SgtPooki/npm-bulk-import-cid that you can use via npm (its at https://www.npmjs.com/package/@sgtpooki/bulk-import-cid) with

$ npx @sgtpooki/bulk-import-cid <filename> <optionalTimeoutMs>

Running it from that repo, with the example.txt file, here is the output:

╰─ ✘ 1 ❯ npx @sgtpooki/bulk-import-cid@latest example.txt
Starting up Kubo daemon...
Reading CIDs from file at path /Users/sgtpooki/code/gist-scripts/bulk-import-cid/example.txt
Read 3 lines from file
daemon stdout: Initializing daemon...
daemon stdout: Kubo version: 0.23.0
daemon stdout: Repo version: 15
daemon stdout: System version: arm64/darwin
daemon stdout: Golang version: go1.21.1
daemon stdout:
daemon stdout: Swarm listening on /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/::1/tcp/4001
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/tcp/4001
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /p2p-circuit
daemon stdout:
daemon stdout: Swarm announcing /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/97.126.55.89/tcp/19055
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip6/::1/tcp/4001
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: RPC API server listening on /ip4/0.0.0.0/tcp/5001
daemon stdout: WebUI: http://0.0.0.0:5001/webui
daemon stdout:
daemon stdout: Gateway server listening on /ip4/127.0.0.1/tcp/8081
daemon stdout:
Kubo daemon is ready
daemon stdout: Daemon is ready
daemon stdout:
Creating import folder to host imported CIDs on MFS at path "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Successfully created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Importing bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq with name "bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Importing QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6 with name "barrel.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Importing QmbvrHYWXAU1BuxMPNRtfeF4DS2oPmo5hat7ocqAkNPr74 with name "pi equals.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Imported all CIDs from "example.txt" to MFS folder "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Shutting down...

and we can see them in IPFS desktop after.

image

NOTE: Since this is adding to the same IPFS blockstore IPFS-Desktop uses, you must shutdown IPFS-Desktop in order to use the above tool, but it should tell you that.

welcome[bot] commented 1 year ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

asigalov61 commented 1 year ago

@SgtPooki Ok. Thank you. I will try it and I will let you know if there are any problems. Thank you again for extra support on this :)

SgtPooki commented 1 year ago

keeping this open as we still would like to support this in ipfs-desktop/ipfs-webui. Moving to ipfs-webui though

asigalov61 commented 1 year ago

@SgtPooki Yes, sorry about closing it. I would love to see the native support for bulk CID import in IPSF desktop :) Thank you, guys, for your hard work. IPFS is an important and needed thing :)

asigalov61 commented 9 months ago

@SgtPooki Dear Sgt Pooki,

Any update on this feature? Any ETA on when it will be implemented?

Thank you so much in advance for your response.

Alex

SgtPooki commented 9 months ago

@asigalov61 unfortunately, we've got a lot of items on our plate now, and a smaller team. I would be happy to review a PR that added a "bulk import" option to the Import button on the files screen:

image

but won't be able to work on this for the forsee-able future.

asigalov61 commented 9 months ago

@SgtPooki Thank you for your response and update :)

Unfortunately, IPFS code is too complex for me to contribute to. Sorry. I am mostly Python ML/DL dev :(

But please add this feature when you are able. I think it would be a useful addition to IPFS.

I will use your workaround for now, for which I am grateful nevertherless :)

Sincerely,

Alex.

itsib commented 5 months ago

Oh, my God. I have the main functionality, file browsing does not work. I click import, and nothing, nothing at all. The interface just ignores everything. No error, no success, nothing. 🥴 I also noticed that react does not run rerender. Most likely, some variable is not in the dependencies of some react hook.

agmap commented 4 months ago

Still waiting for bulk import of many CIDs from IPFS via csv or txt file. If it is available via the UI?

MattWong-ca commented 3 weeks ago

Hey @SgtPooki, interested in working on this! To confirm in terms of UX, would the below flow work?

Click 'Import' --> Choose 'Bulk Import' (new option) --> user selects .txt or .csv file --> new imported files show up on the page

asigalov61 commented 3 weeks ago

@SgtPooki @MattWong-ca Thank you very much for wanting to add this much needed feature! Looking forward to the final implementation :)

agmap commented 3 weeks ago

Many thanks in advance. I really miss such a function. Perhaps a bulk import of locally stored CAR files can also be integrated?