CalebEverett / arloader

Rust command line application and client for uploading files to Arweave.
Apache License 2.0
82 stars 23 forks source link

NotFound when using multiplier 0 to 3 #19

Closed mccode415 closed 2 years ago

mccode415 commented 2 years ago

bundle txid items KB status confirms

fkESPupWBD-TVpXaBGJC-ta7--UDjtdiJ-Uap2WMSEo 4000 3173 NotFound 0 Updated 1 statuses.

Is there any work around for this issue or anyway to get the sol back?

CalebEverett commented 2 years ago

Yes, just let me know the sol tx, or the address and I can refund. Did you get any other error messages?

CalebEverett commented 2 years ago

Also can you let me know the number and size of the files you were attempting to upload or paste the command here that didn't work?

CalebEverett commented 2 years ago

This one looks like it is uploaded: https://viewblock.io/arweave/tx/fkESPupWBD-TVpXaBGJC-ta7--UDjtdiJ-Uap2WMSEo.

The gateway is still taking a long time to index the bundles. If you have a smaller number of files you can upload without bundles by passing --no-bundle if you are using the upload command.

mccode415 commented 2 years ago

arloader upload ./updated/*.json --with-sol --ar-default-keypair --sol-keypair-path ~/.config/solana/key.json

4000 files all json

mccode415 commented 2 years ago

Hmmm looks like it has a few confirmation is that enough? I tried to create a manifest but none of the urls are working.

arloader get-status U1Jxusym_e20NhwV10vgE43cfHfDHORGw4GRSuhp6Cs
 id                                           status     confirms
------------------------------------------------------------------
 U1Jxusym_e20NhwV10vgE43cfHfDHORGw4GRSuhp6Cs  Confirmed        49

Here's an example

"./updated/0.json": {
        "id": "eYxlhAuT3kDAqLPW1EmNikiN__brY3zXrinUiqvosvo",
        "files": [
            {
                "uri": "https://arweave.net/eYxlhAuT3kDAqLPW1EmNikiN__brY3zXrinUiqvosvo",
                "type": "application/json"
            },
            {
                "uri": "https://arweave.net/U1Jxusym_e20NhwV10vgE43cfHfDHORGw4GRSuhp6Cs/./updated/0.json",
                "type": "application/json"
            }
        ]
    }
mccode415 commented 2 years ago

@CalebEverett how are you able to tell the gateway is taking a really long time?

CalebEverett commented 2 years ago

@mccode415 If you look on viewblock https://viewblock.io/arweave/address/7nbOsbMYb86qN1YSS0FM514DBctzJ1Swq1PiuXJEWsk and click on the purple icons, when they have been parsed you will see the files in the bundle when you click on it. Same thing if you query graphql https://arweave.net/graphql, you will see the bundle items once the gateway has parsed it, and just the bundles by themselves when it hasn't yet.

Here's one from 22 hours ago, that has been parsed: https://viewblock.io/arweave/tx/5ii-5-lC8T2FV5-5WoErmh3_8ZO5I77HkZVXl3Vejzs.

And here is one from 11 hours ago that has not been parsed. https://viewblock.io/arweave/tx/fkESPupWBD-TVpXaBGJC-ta7--UDjtdiJ-Uap2WMSEo.

Arweave discord was saying that it had not caught back up completely from being down earlier this week.

I think if you look at the overall transaction log, you will find the same thing. Here is a bundle from bundlr from five hours ago that also isn't parsed yet. https://viewblock.io/arweave/tx/Rhrl0fwo7GVfjSK2NCYBAHAJ0PVFPi_8nXEKuFlX_TE

CalebEverett commented 2 years ago

Also, I think the manifest files require parsing by the gateway as well - not sure if the backlog on them is the same as for the bundles, but still seems to be taking a while. A couple of days ago, it was like three days back, so if it is between 10 and 20 hours behind now, hopefully that means it should be caught back up soon. Before it went down, it was pretty fast - usually minutes.

CalebEverett commented 2 years ago

And I think the number I got from Arweave discord in terms of numbers of confirmations, was 25 to be virtually certain there won't be a fork that leaves the block with your transaction behind.

mccode415 commented 2 years ago

I see I see so basically, we need to wait for all 4000 files to appear in bundled transaction area before creating a manifest? Or is the current manifest still ok even though the files have not been fully indexed?

CalebEverett commented 2 years ago

Current manifest should be fine. The ids of the items in the bundle are deterministic - so the ids in the manifest are good, it is just a matter of the gateway parsing it so that when you request by those ids, it has the content to be able to serve. If for some reason the bundle doesn't get confirmed or there is a problem with the parsing that meant you needed to reupload, then you would need to create a new manifest. But otherwise, unfortunately, it is just a matter of time.

CalebEverett commented 2 years ago

It looks like these parsed: https://.arweave.net/eYxlhAuT3kDAqLPW1EmNikiN__brY3zXrinUiqvosvo

mccode415 commented 2 years ago

@CalebEverett thanks for keeping with this issue I really appreciate it! Looks like manifest mapping is working too from the link I posted above.