ipfs.add(stream) 504 gateway timeout

[SourceBoy]

• Version: 0.18.0

  {
  version: '0.18.0',
  commit: '',
  repo: '12',
  'ipfs-core': '0.18.0',
  'interface-ipfs-core': '^0.158.0'
  }

• Platform: amd64 (docker node:18-slim)

• Subsystem: ipfs-core

Severity: High

Description:

ipfs.add(<stream>) receives a cid but gets a 504 during content retrieval. Doc ref: https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#filecontent

Steps to reproduce the error:

import * as IPFS from 'ipfs-core';

// const stream = Neither WHATWG ReadableStream nor Node stream.Readable works

(async () => {
  const ipfs = await IPFS.create();
  const result = await ipfs.add(stream);
  console.log(result);
})();

[welcome[bot]]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[SgtPooki]

Hi there!

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and the corresponding migration guide here.

It seems like the core issue you're running into is that you are getting a 504 "gateway timeout" when attempting to retrieve your content, but you haven't really specified how you're trying to retrieve your content. This could be due to not "publishing" the data you created/added on ipfs, or downloading too much content from the gateway you're using, etc.

This current bug or feature request will not be addressed in js-ipfs due to its deprecation.

As a result, we are going to close this issue. If you think we have done this in error, please feel to reopen with any comments in the next week as we will circle back on the reopened issues.

We hope you will consider Helia for your IPFS in JS needs. If you believe this particular request belongs in Helia, feel free to open a Helia issue. We look forward to engaging with you more there.

Thanks, @ipfs/helia-dev

[SourceBoy]

Content was retrieved from https://ipfs.io/ipfs/<cid> as shown from the doc section.

The .add() call did return a cid back.

Does @helia/unixfs support adding a WHATWG ReadableStream or Node stream.Readable? Our use case is to take a http stream upload and then stream-upload the file to IPFS without buffering the file in memory or storing the file on file system.

Thanks.

[SgtPooki]

@SourceBoy it looks like unix-fs supports adding content via ImportCandidateStream (https://github.com/ipfs/helia-unixfs/blob/933179c028805868dcf5bd0a936210eb6d653db9/packages/unixfs/src/commands/add.ts#L21) and ByteStream(https://github.com/ipfs/helia-unixfs/blob/933179c028805868dcf5bd0a936210eb6d653db9/packages/unixfs/src/commands/add.ts#LL37C45-L37C55). You can see the tests at https://github.com/ipfs/helia-unixfs/blob/main/packages/unixfs/test/add.spec.ts.

https://www.npmjs.com/package/ipfs-unixfs-importer is the package that @helia/unixfs is using with the addAll and addBytes methods. They accept an ImportCandidateStream and ByteStream respectively. Check out https://github.com/ipfs/js-ipfs-unixfs/tree/master/packages/ipfs-unixfs-importer#api for details on what's going on behind the scenes.

[SourceBoy]

@SgtPooki Thank you

[SgtPooki]

No problem! Please let us know with a an issue on the helia/unixfs repo if helia/unixfs doesn't work for your usecase and we’ll work to help you out there :)