search

textileio / powergate

Multitiered file storage API built on Filecoin and IPFS
MIT License
385 stars 93 forks source link

I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

Open ChristianOConnor opened 3 years ago

ChristianOConnor commented 3 years ago

I made a hot storage deal and I still can't access the content on IPFS. The content that I put into this deal is the directory of a static website. When I double click the index.html file in Windows Explorer, it opens perfectly in Firefox so I know the website works. I tried to access it with https://ipfs.io/ipfs/<CID> and https://<CID>.ipfs.dweb.link but they both failed. I did the deal perfectly. I did it exactly to spec for a hot storage deal which I believe is the way to get your content on IPFS. Here's what I did to make my storage deal.

My config file aka config-new.json looks like this. 

{
  "hot":  {
    "enabled":  true,
    "allowUnfreeze":  true,
    "unfreezeMaxPrice":  "100000000000000000",
    "ipfs":  {
      "addTimeout":  "900"
    }
  },
  "cold":  {
    "enabled":  true,
    "filecoin":  {
      "replicationFactor":  "2",
      "dealMinDuration":  "518400",
      "excludedMiners":  [],
      "trustedMiners":  ["<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>",],
      "countryCodes":  [],
      "renew":  {
        "enabled":  false,
        "threshold":  "0"
      },
      "address":  "<my filecoin address>",
      "maxPrice":  "100000000000000000",
      "fastRetrieval":  true,
      "dealStartOffset":  "8640",
      "verifiedDeal":  false
    }
  },
  "repairable":  false
}

1) git clone https://github.com/textileio/powergate.git
2) cd powergate/
3) cd docker/
4) nano docker-compose.yaml where I added "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" between lines 32 and 33.
5) make up
6) Waited for the node to finish importing and then syncing.
7) ^C then make down then deleted the line "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" from docker-compose.yaml
8) make up
9) Now that the node was running I typed cd .. so I was in the repo's root directory, then make install-pow
10) with the pow command in my GOPATH I typed pow to make sure pow was linked fine to powd. It was.
11) pow admin users create
12) copied the token and ran export POW_TOKEN=<token copied to here>
13) Then pow wallet addrs and funded the address
14) I went to the directory behind the folder of my static website which is about 5GB in size.
15) I typed pow data stage <my-static-site-folder>
16) After it was finished staging and printed out the CID I typed pow config apply --watch <CID> -c custom-080121-hot-config.json waited a long time and it worked! I had a Filecoin deal that was duplicated across 2 miners.

So I now have a storage deal but the directory still isn't on IPFS. Why won't it work?

ChristianOConnor commented 3 years ago

@jsign

ChristianOConnor commented 3 years ago

@jsign @andrew This is still a pressing issue, and at this point, I think it's a bug in the code and not something that I did. My files still cannot be found on IPFS. Please someone explain to me what the problem is. What is going on?

andrewxhill commented 3 years ago

it sounds like you are running this in a way where your IPFS node isn't accessible to the IPFS network.

here is a good discussion w/ different checks & solutions https://discuss.ipfs.io/t/added-a-file-but-its-not-accessible-on-gateway/4667/5

ChristianOConnor commented 3 years ago

it sounds like you are running this in a way where your IPFS node isn't accessible to the IPFS network.

here is a good discussion w/ different checks & solutions https://discuss.ipfs.io/t/added-a-file-but-its-not-accessible-on-gateway/4667/5

Awesome thanks for the info I'm spinning up my Powergate node now to troubleshoot.
Still though, as I'm waiting for my Powergate node to sync, doesn't it not really matter if IPFS can't connect? Doesn't it deploy to IPFS from the Filecoin miner? I though hot storage meant that Filecoin stores the file and deploys it to IPFS.

ChristianOConnor commented 3 years ago

Now I can't access my install anymore. I had to move my install to another computer and now the lotus node just says this: 

lotus_1       | 2021-08-30T08:55:51.357Z        INFO    backupds        backupds/log.go:125         opening log     {"file": "/data/.lotus/kvlog/metadata/1630310485.log.cbor"}
lotus_1       | ERROR: initializing node: starting node: could not build arguments for function     "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/    lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".    makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.RawHost: could not     build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12):     could not build value group []config.Option[group="libp2p"]: could not build arguments for     function "github.com/filecoin-project/lotus/node/modules/lp2p".ConnGaterOption (/lotus/node/    modules/lp2p/conngater.go:14): failed to build *conngater.BasicConnectionGater: could not build     arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed     to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/    local/go/src/reflect/asm_amd64.s:12): opening backupds: opening log: loghead log file doesn't     match, opening /data/.lotus/kvlog/metadata/1630310485.log.cbor, expected 1630206194.log.cbor
lotus_1       | 
mainnet_lotus_1 exited with code 1

Every time I try to fix a problem with this I just get more problems lol. Does anyone know what this error means?

jsign commented 3 years ago

The Lotus node can be somewhat complex to operate and I've deal (and still do) with some errors like that one. To be clear, that's not particularly related to Powergate but is more like a Lotus question.

For that error you can:

  1. Stop lotus
  2. Remove /data/.lotus/kvlog/metadata/1630310485.log.cbor, this file should be safe to delete. (you'll have to believe me on this; might worth making a backup)
  3. Start again lotus.
ChristianOConnor commented 3 years ago

The Lotus node can be somewhat complex to operate and I've deal (and still do) with some errors like that one. To be clear, that's not particularly related to Powergate but is more like a Lotus question.

For that error you can:

1. Stop lotus

2. Remove `/data/.lotus/kvlog/metadata/1630310485.log.cbor`, this file should be safe to delete. (you'll have to believe me on this; might worth making a backup)

3. Start again lotus.

Okay I did what you said. I went to the docker directory of the repo then make up. I opened a new terminal and went into the lotus node with docker exec -it mainnet_lotus_1 /bin/sh then real quick ran the command rm /data/.lotus/kvlog/metadata/1630310485.log.cbor. Then I exited out of the docker container.

Now I get the following error:

lotus_1       | 2021-08-30T20:33:16.287Z        ERROR   statetree       state/statetree.go:266  failed to load state tree: failed to load hamt node: blockstore: block not found

I restarted the node and now I got this more complete error 

lotus_1       | 2021-08-30T20:40:02.531Z        WARN    modules modules/services.go:72  failed to     say hello     {"error": "load state tree: failed to load state tree: failed to load hamt node:     blockstore: block not found", "errorVerbose": "load state tree:\n    github.com/filecoin-project/    lotus/chain/store.(*ChainStore).Weight\n        /lotus/chain/store/weight.go:34\n  - failed to     load state tree:\n    github.com/filecoin-project/lotus/chain/state.LoadStateTree\n        /lotus/    chain/state/statetree.go:267\n  - failed to load hamt node:\n    github.com/filecoin-project/    specs-actors/actors/util/adt.AsMap\n        /go/pkg/mod/github.com/filecoin-project/    specs-actors@v0.9.13/actors/util/adt/map.go:41\n  - blockstore: block not found", "peer":     "12D3KooWNtDE3J7pdtAzq6Rp3d8eHnJ663H6EpUYxr3xBaMTJRhJ", "supported": ["/ipfs/id/1.0.0", "/meshsub/    1.0.0", "/fil/chain/xchg/0.0.1", "/chain/ipfs/bitswap/1.0.0", "/p2p/id/delta/1.0.0", "/floodsub/1.    0.0", "/chain/ipfs/bitswap/1.1.0", "/fil/datatransfer/1.1.0", "/fil/datatransfer/1.0.0", "/ipfs/    ping/1.0.0", "/libp2p/autonat/1.0.0", "/fil/hello/1.0.0", "/fil/sync/blk/0.0.1", "/chain/ipfs/    bitswap", "/ipfs/id/push/1.0.0", "/meshsub/1.1.0", "/chain/ipfs/bitswap/1.2.0", "/ipfs/graphsync/1.    0.0", "/fil/kad/testnetnet/kad/1.0.0"], "agent": "lotus-1.10.1+mainnet+git.d6f3d3c36"}
jsign commented 3 years ago

That's just a warning which is pretty common. How are you evaluating if things are working properly or not?

Like, does: docker exec -it testnet_lotus_1 lotus sync wait say that you're syncing and all is OK?

ChristianOConnor commented 3 years ago

Oh nice, yeah It looks like it's working. This is the result. 

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 10; Base: 1064014; Target: 1068500 (diff: 4486)
State: fetching messages; Current Epoch: 1064013; Todo: 4487
Validated 0 messages (0 per second)
ChristianOConnor commented 3 years ago

Wtf... now the messages are making no sense 

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 15; Base: 1064014; Target: 1068523 (diff: 4509)
State: header sync; Current Epoch: 1068523; Todo: 0
Validated 0 messages (0 per second)

the diff is higher but Todo is 0????

ChristianOConnor commented 3 years ago

Now the number is going up 

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 15; Base: 1064014; Target: 1068523 (diff: 4509)
State: fetching messages; Current Epoch: 1064013; Todo: 4510
Validated 0 messages (0 per second)
jsign commented 3 years ago

Diff should be going down. Considering your 4k TODO you definitely want to sync from a snapshot again. https://docs.textile.io/powergate/mainnet/#recover-or-prune-your-existing-lotus-node-with-a-snapshot

After the import you'll be ~300 blocks await from the tip.

ChristianOConnor commented 3 years ago

Nope, still not working. I re-sunk the lotus node following your link. It still won't sync. The diff just keeps growing and not syncing a single block.

Here it is at 10:47PM EST
docker exec -it mainnet_lotus_1 lotus sync wait Worker: 3; Base: 1064014; Target: 1069049 (diff: 5035) State: fetching messages; Current Epoch: 1064013; Todo: 5036 Validated 0 messages (0 per second)

And here it is at 10:57PM EST
docker exec -it mainnet_lotus_1 lotus sync wait Worker: 4; Base: 1064014; Target: 1069063 (diff: 5049) State: error; Current Epoch: 1064013; Todo: 5050 Validated 0 messages (0 per second)

I did the re-sync exactly as your link said.
1) make down
2) docker run --rm -v mainnet_powergate-lotus:/data ubuntu rm -rf /data/.lotus/datastore/chain
3) add command: ["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"] between line 32 and 33 of docker-compose.yaml.
4) make up
5) wait for the sync to finish (and I waited an extra 10 - 20 minutes after the blockchain was downloaded to make sure the sync completed).
6) ^C the docker stack
7) make down
8) docker run --rm -v mainnet_powergate-lotus:/data ubuntu rm -rf /data/.lotus/datastore/chain
9) deleted line 33 added in step 3 to the docker-compose.yaml file
10) make up

And it's still not syncing. the Diff just keeps getting bigger... wtf... ????!!!!

ChristianOConnor commented 3 years ago

I tried the 10 steps again and now I get this error at step 4:
ERROR: initializing node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.NetworkName: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.AfterGenesisSet: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): getting genesis block failed: blockstore: block not found

jsign commented 3 years ago

Most probably some step above wasn't executed correctly. (e.g: yesterday I pruned some nodes doing exactly that).

ChristianOConnor commented 3 years ago

Most probably some step above wasn't executed correctly. (e.g: yesterday I pruned some nodes doing exactly that).

* Did step 4 download a 40GiB file? Time between 4 and 5 should be at least 20 or 30 min today if you have very good bandwidth.

* After doing step 2, you can `docker run` inside the container after and check that `/data/.lotus/datastore/chain` folder doesn't exist (or is very small if was boostraped again when you started the container). If this isn't the case, your Lotus volume might not be `mainnet_powergate-lotus`.

Now it liturally won't even let me start the node. I get this now...

"lotus_1 | ERROR: initializing node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.NetworkName: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.AfterGenesisSet: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): getting genesis block failed: blockstore: block not found"

This is beyond frustrating...

jsign commented 3 years ago

I'm sorry to hear that. I hope solving this Lotus problem can be easier, but usually deleting the state and spinning again from a snapshot usually solves all the problems.

ChristianOConnor commented 3 years ago

I'm sorry to hear that. I hope solving this Lotus problem can be easier, but usually deleting the state and spinning again from a snapshot usually solves all the problems.

It's not your fault and I'm really sorry to be such a pain with this. I just really want to incorporate Filecoin as the backend of my storage app and for some reason that's proving to be impossible. I will file a bug request with Lotus to see what they say.

Again I really appreciate your help and I love all that you guys do. Please keep following this issue thread as I continue to post updates and ask for help regarding this situation. I really appreciate that you guys have been helping me along the way.

jsign commented 3 years ago

We're working on a new and easier solution to allow people like you use Filecoin for your products. I'll keep you posted around that.

ChristianOConnor commented 3 years ago

We're working on a new and easier solution to allow people like you use Filecoin for your products. I'll keep you posted around that.

Oh wow that would be amazing! Please keep me posted on that. You can reach me through Github or contact me on Reddit as "u/warpanomaly" or Twitter as "OConnorROCKS" Thanks again!