search

ipfs / js-ipfs

IPFS implementation in JavaScript
https://js.ipfs.tech
Other
7.44k stars 1.25k forks source link

ipfs.dag.put HTTPError: missing an unmarshaller for tag 1 when using ipfs-http-client #3043

Closed stanbar closed 1 year ago

stanbar commented 4 years ago

Severity:

High - The main functionality of the application does not work, API breakage, repo format

Description:

I want to add to dag the object that looks like this:

const voting = {
    "title": "asdfasd",
    "polls": [
        {
            "_id": "5ec46e322d7e49029617fa2e",
            "question": "question",
            "options": [
                {
                    "_id": "5ec46e322d7e49029617fa2f",
                    "name": "one",
                    "code": 1
                },
                {
                    "_id": "5ec46e322d7e49029617fa30",
                    "name": "two",
                    "code": 2
                }
            ]
        }
    ],
    "issueAccountId": "GCENXCYETHIROKE22XBUPVGRW7FAQJVN32VCH4XKXEIFJMYFA532HSVI",
    "assetCode": "Newvoting",
    "distributionAccountId": "GAWSMPQUG3YK4ED6YVVQKWJ4FJQ5FBS3TTCI3TWSQVSML7RWEOUJB7U6",
    "ballotBoxAccountId": "GBOFKKWIEBFOWQ23UOUSYZOW6TFMGHFJD6PXKWU725TAX724AE5KBJV6",
    "authorization": "open",
    "visibility": "public",
    "votesCap": 100,
    "encryption": {
        "encryptionKey": "MDgCEQDPE5eb9/3Qz7cJqUQuiilnAhBE+ZH2pLxJOAii2tzKbZyIAhEAn7UjCy+yLapZs6Nr0GxRgQ==",
        "encryptedUntil": "2020-05-27T23:16:27.858Z"
    },
    "startDate": "2020-05-19T23:16:26.689Z",
    "endDate": "2020-05-27T23:16:27.858Z",
    "slug": "voting-vgmxmt",
    "createdAt": "2020-05-19T23:39:30.912Z",
    "updatedAt": "2020-05-19T23:39:30.912Z",
    "id": "5ec46e322d7e49029617fa2d"
}

when i call the function 

const cid = (await ipfs.dag.put(voting, { pin: true, format: 'dag-cbor', hashAlg: 'sha2-256' })).toString()

it throws 

 HTTPError: missing an unmarshaller for tag 1
     at Object.errorHandler [as handleError] (/server/node_modules/ipfs-http-client/src/lib/core.js:67:15)
     at processTicksAndRejections (internal/process/task_queues.js:97:5)
     at Client.fetch (/server/node_modules/ipfs-utils/src/http.js:155:9)
     at Object.put (/server/node_modules/ipfs-http-client/src/dag/put.js:46:17)

I'm using ipfs-http-client and the actual node is running in docker container.

I looked at the implementation and the serializer seems to be working fine

import * as dagCBOR from 'ipld-dag-cbor'

const voting = // { ... }
const serialized = dagCBOR.util.serialize(voting)

Is working fine.

achingbrain commented 4 years ago

I'm having problems replicating this. I've tried creating a minimal project:

package.json:

{
  "dependencies": {
    "go-ipfs-dep": "^0.5.1",
    "ipfs-http-client": "^44.1.0",
    "ipfsd-ctl": "^4.1.1"
  }
}

index.js:

const { createFactory } = require('ipfsd-ctl')

async function main () {
  const factory = createFactory({
    type: 'go',
    test: true,
    ipfsHttpModule: require('ipfs-http-client'),
    ipfsBin: require('go-ipfs-dep').path()
  })
  const node = await factory.spawn()
  const voting = {
    // copied from above
  }
  const cid = await node.api.dag.put(voting, { pin: true, format: 'dag-cbor', hashAlg: 'sha2-256' })
  console.info(`put ${cid}`)
  await node.stop()
}

main()
  .catch(err => {
    console.error(err)
    process.exit(1)
  })

console:

$ node index.js
put bafyreidccdo62shnbnjvcbmqunrvqglckydhjp7zzu3n4csfeagwqbfv3q

ipfs-http-client runs it's interface tests against go-ipfs - could you add a test to the dag.put suite that fails with the same error you are seeing?

To run the tests:

$ git clone https://github.com/ipfs/js-ipfs.git
$ cd js-ipfs
$ npm i
// now edit packages/interface-ipfs-core/src/dag/put.js and add your test
$ cd packages/ipfs-http-client
$ npm run test:node

When it's failing, please create a PR to this repo with your changes.

stanbar commented 4 years ago

I wrote the test, and it's passing. So I guess the problem is somewhere else. The error throws from HTTP request, so maybe the go-ipfs node (running in docker), doesn't understand the serialized content. The thing is, when I put something else like:

await node.api.dag.put({ hello: 'world' }, { pin: true, format: 'dag-cbor', hashAlg: 'sha2-256' })

it is working fine.

I've found the workaround, if I first stringify and parse the object it gets added successfully.

const normalizedObject = JSON.parse(JSON.stringify(voting))
await ipfs.dag.put(normalizedObject, { pin: true, format: 'dag-cbor', hashAlg: 'sha2-256' }

I guess there is some problem with the voting object itself (I'm fetching it from mongodb). So you can close the issue, if you don't know why is that happening. Thank you for your time.

achingbrain commented 4 years ago

Cbor tag 1 is for dates so I wonder if something weird is going on there. This error is coming from the remote node though - any ideas @Stebalien?

achingbrain commented 4 years ago

I’m guessing startDate/endDate etc are actually Date objects in memory and not strings as above.

achingbrain commented 4 years ago

It's definitely that, if I change:

"createdAt": "2020-05-19T23:39:30.912Z",
"updatedAt": "2020-05-19T23:39:30.912Z",

to:

"createdAt": new Date("2020-05-19T23:39:30.912Z"),
"updatedAt": new Date("2020-05-19T23:39:30.912Z"),

I get:

$ node index.js
HTTPError: missing an unmarshaller for tag 1
    at Object.errorHandler [as handleError] (/Users/alex/test/bar/node_modules/ipfs-http-client/src/lib/core.js:67:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Client.fetch (/Users/alex/test/bar/node_modules/ipfs-utils/src/http.js:155:9)
    at async Object.put (/Users/alex/test/bar/node_modules/ipfs-http-client/src/dag/put.js:52:17)
    at async main (/Users/alex/test/bar/index.js:49:15) {
  name: 'HTTPError',
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'http://127.0.0.1:54141/api/v0/dag/put?format=dag-cbor&input-enc=raw&pin=true&hash=sha2-256',
      status: 500,
      statusText: 'Internal Server Error',
      headers: [Headers],
      counter: 0
    }
  }
}

js-IPFS doesn't throw the same error, so I suppose there's only a subset of CBOR tags that go-IPFS supports?

achingbrain commented 4 years ago

@Stebalien any thoughts on the above? Looks like go-ipfs doesn't support unmarshalling cbor with tags that aren't CIDs or simple strings/numbers/maybe?

achingbrain commented 1 year ago

js-ipfs is being deprecated in favor of Helia. You can https://github.com/ipfs/js-ipfs/issues/4336 and read the migration guide.

You can swap ipfs-http-client for kubo-rpc-client - it's a drop-in replacement and where compatibility fixes with Kubo will get added in future.