search

ipfs / ipfs-webui

A frontend for an IPFS Kubo and IPFS Desktop
https://webui.ipfs.io
MIT License
1.55k stars 482 forks source link

feat: bulk CIDs import #2173

Open asigalov61 opened 11 months ago

asigalov61 commented 11 months ago

Hey guys,

I was wondering if it would be possible to add a bulk files/CIDs import from a txt or csv file?

I have a list of CIDs but there is no option in IPFS-Desktop to import them in bulk/automatically.

Can you implement this option? Or, if it is possible to do via Kubo or some other script method, it would be nice to see such method in the docs.

Please let me know.

Alex

welcome[bot] commented 11 months ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

SgtPooki commented 11 months ago

@asigalov61 Great question. You should be able to do this very easily by dragging and dropping files onto the IPFS-Desktop "Files" screen. Please let us know if you experience any issues with this.

asigalov61 commented 11 months ago

@SgtPooki Thank you for your response. But I was asking about download, not upload.

I have a csv file with IPFS CIDs and I want to import all CIDs so that I can download them.

There seems to be only option to do it one by one by going to Files -> Import from IPFS, but there is no bulk "Import from IPFS" option.

Am I missing something?

Please let me know.

Thank you.

github-actions[bot] commented 10 months ago

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

SgtPooki commented 10 months ago

@asigalov61 I see! sorry for my confusion.. I think an option to bulk import CIDs would be a great addition to IPFS-Desktop. Just FYI: that change would likely live in the ipfs-webui repo.

It might be a while until we get to this feature since a lot of our focus has been on Helia, but to unblock yourself for now, I threw together https://github.com/SgtPooki/npm-bulk-import-cid that you can use via npm (its at https://www.npmjs.com/package/@sgtpooki/bulk-import-cid) with

$ npx @sgtpooki/bulk-import-cid <filename> <optionalTimeoutMs>

Running it from that repo, with the example.txt file, here is the output:

╰─ ✘ 1 ❯ npx @sgtpooki/bulk-import-cid@latest example.txt
Starting up Kubo daemon...
Reading CIDs from file at path /Users/sgtpooki/code/gist-scripts/bulk-import-cid/example.txt
Read 3 lines from file
daemon stdout: Initializing daemon...
daemon stdout: Kubo version: 0.23.0
daemon stdout: Repo version: 15
daemon stdout: System version: arm64/darwin
daemon stdout: Golang version: go1.21.1
daemon stdout:
daemon stdout: Swarm listening on /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/::1/tcp/4001
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/tcp/4001
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /p2p-circuit
daemon stdout:
daemon stdout: Swarm announcing /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/97.126.55.89/tcp/19055
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip6/::1/tcp/4001
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: RPC API server listening on /ip4/0.0.0.0/tcp/5001
daemon stdout: WebUI: http://0.0.0.0:5001/webui
daemon stdout:
daemon stdout: Gateway server listening on /ip4/127.0.0.1/tcp/8081
daemon stdout:
Kubo daemon is ready
daemon stdout: Daemon is ready
daemon stdout:
Creating import folder to host imported CIDs on MFS at path "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Successfully created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Importing bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq with name "bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Importing QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6 with name "barrel.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Importing QmbvrHYWXAU1BuxMPNRtfeF4DS2oPmo5hat7ocqAkNPr74 with name "pi equals.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Imported all CIDs from "example.txt" to MFS folder "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Shutting down...

and we can see them in IPFS desktop after.

image

NOTE: Since this is adding to the same IPFS blockstore IPFS-Desktop uses, you must shutdown IPFS-Desktop in order to use the above tool, but it should tell you that.

welcome[bot] commented 10 months ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

asigalov61 commented 10 months ago

@SgtPooki Ok. Thank you. I will try it and I will let you know if there are any problems. Thank you again for extra support on this :)

SgtPooki commented 10 months ago

keeping this open as we still would like to support this in ipfs-desktop/ipfs-webui. Moving to ipfs-webui though

asigalov61 commented 10 months ago

@SgtPooki Yes, sorry about closing it. I would love to see the native support for bulk CID import in IPSF desktop :) Thank you, guys, for your hard work. IPFS is an important and needed thing :)

asigalov61 commented 7 months ago

@SgtPooki Dear Sgt Pooki,

Any update on this feature? Any ETA on when it will be implemented?

Thank you so much in advance for your response.

Alex

SgtPooki commented 7 months ago

@asigalov61 unfortunately, we've got a lot of items on our plate now, and a smaller team. I would be happy to review a PR that added a "bulk import" option to the Import button on the files screen:

image

but won't be able to work on this for the forsee-able future.

asigalov61 commented 6 months ago

@SgtPooki Thank you for your response and update :)

Unfortunately, IPFS code is too complex for me to contribute to. Sorry. I am mostly Python ML/DL dev :(

But please add this feature when you are able. I think it would be a useful addition to IPFS.

I will use your workaround for now, for which I am grateful nevertherless :)

Sincerely,

Alex.

itsib commented 3 months ago

Oh, my God. I have the main functionality, file browsing does not work. I click import, and nothing, nothing at all. The interface just ignores everything. No error, no success, nothing. 🥴 I also noticed that react does not run rerender. Most likely, some variable is not in the dependencies of some react hook.

agmap commented 1 month ago

Still waiting for bulk import of many CIDs from IPFS via csv or txt file. If it is available via the UI?