search

CalebEverett / arloader

Rust command line application and client for uploading files to Arweave.
Apache License 2.0
83 stars 23 forks source link

Insanely high upload time? #15

Closed nprodas closed 2 years ago

nprodas commented 2 years ago

Hey team!

Was not sure where the best place to put this was, but figured it couldn't hurt to have it here in case any other users experience this.

My scenario: Our project already minted and we want to add an extra attributes field in the metadata to specify the rarity of that NFT. This needs to be done for all 750 NFT's and we can easily update the URI that points to the metadata with something like metaboss. The problem we face is figuring out the best way to upload the updated JSONS into arweave.

The issue: I did some testing with only 1 JSON file following the available documentation. I used SOL and also specified the maximum amount for reward multiplier. Even with the increased rewards, the upload for a single JSON file took roughly 40 minutes before it showed up.

Is there something I am doing wrong for it to take so long? Is there a better way to achieve what we are trying to do?

Any help would be greatly appreciated!

CalebEverett commented 2 years ago

Feel free to dm me on discord if you want some real time help: @Caleb#9071.

That sounds like a long time, but if it ultimately showed up, it could be a gateway issue. Depending on the size of your files, there are a several different things we could try to get things up there faster.

For starters, make sure you are on the latest version: 0.1.59. That should have a max reward multiplier of 3 and with just 750 json files should just be a single bundle.

nprodas commented 2 years ago

Hey, I sent you a friend request on discord, my user name is @NoRelax#5788

CalebEverett commented 2 years ago

Hi @nprodas I didn't see that in discord, but happy to help. Are you still experiencing delays? I think the arweave gateway was behind today.

paul-hl commented 2 years ago

I also sent you a friend request. If I may, I would like to discuss upload times as well. It seems slow.. thank you

CalebEverett commented 2 years ago

Hi @paul-hl I didn't get that one either, but what seems to be the trouble? Are you having issues with the time it takes to complete the initial upload of files or time it takes before they are confirmed and/or available?

CalebEverett commented 2 years ago

I think the markdown was getting my discord address with a lower case c instead of upper case.

@Caleb#9071

CalebEverett commented 2 years ago

One other suggestion generally is that sometimes it seems the indexing of the bundles can take some time on the network, so if you have only a small number of files to upload, <100 or so, you can pass the --no-bundle flag and they will be uploaded as individual transactions, not bundled together. They will then be available without the network having to index any bundles.

paul-hl commented 2 years ago

Thanks for the suggestion. I had arrived to this solution myself as well but I am glad to know why it is quicker (and better in my case).

And I meant that it is so slow not to upload the image itself, but it takes a lot of time before it is actually viewable when using upload-nft.

I just need to upload single images, so no-bundle is a great option for me.

However, I am getting Submitted and after a few seconds NotFound status of my transaction today. I have two questions about that. I am using --reward-multiplier 3.0. It came through to Confirmed just once this afternoon.

My questions:

  1. I can see that the SOL was transferred. So it means, when the transaction is rejected by Arweave, they still keep the fee? :-D
  2. Is it often that the tx is rejected (NotFound)?
  3. I can see the image even if the tx is NotFound but I am not sure it will stay available after some time.

Thanks.

CalebEverett commented 2 years ago

@paul-hl I think the gateway has been having issues with the bundles the last couple of days. I am checking to make sure it is not an arloader issue, but I have been hearing of other people experiencing delays as well - I think especially with bundles and manifests. Try starting with 2 on the reward multiplier, sometimes they can get rejected for too much reward. If the SOL was transferred and your arweave tx failed, send me a list of tx and I can send it back to you - sorry it is manual right now. And that is right, you can't rely on being able to see the image as confirmation it has been permanently added - you have to relay on number of confirmations and being able to retrieve the data from https://aweave.net/tx/{txid} or https://aweave.net/chunk/.

Also sometimes it can be useful to check in over at the arweave discord channel just to see if other people are experiencing issues because there is something going on with the network. I just looked and it looks like there are network issues.

paul-hl commented 2 years ago

@CalebEverett thanks.

I tried reward 2, 2.5, still getting NotFound multiple times or sometimes I get SolanaNetworkError.

Do you think using AR token for payment, or alternatively the arbk tool from arweave, is more reliable? I prefer paying in SOL, but..

CalebEverett commented 2 years ago

I can try here to see if I am having the same issues.

CalebEverett commented 2 years ago

@paul-hl I was able to upload single transactions ok here with AR. I was having trouble with SOL, which means that the SOL network is taking longer than 10 seconds to finalize transactions. When that happens, you can wait until later or try using AR.

paul-hl commented 2 years ago

@CalebEverett thank you for trying this out.

CalebEverett commented 2 years ago

@paul-hl no worries - let me know if you need a little AR to tide you over - getting it can be a pain.

It can be tough to figure out exactly why transactions aren't going through as fast as they should or at all, unfortunately, given all the moving pieces, especially with paying with SOL. You've got arloader, solana and arweave (which depends on both the gateway for uploading and indexing and the nodes for actually writing to the chain), all in the mix and they all have to be working in order for things to go smoothly. This or dm on discord is the right place for arloader.

For arweave, you can run arloader pending to check the pending transaction volume - if that is over 1,000 then it could be a congestion issue. There are also potential gateway issues, but I don't think there is a gateway status site, so you have to check the arweave discord. You can also try https://scar.vercel.app/ or https://viewblock.io/arweave. If you see big lags in the time since the last transaction was created or you see bundle transactions that are not being indexed to show the included data items older than an hour or so, that usually means something is up.

Here is one where the indexing was completed from two days ago: https://viewblock.io/arweave/tx/stWzggrjBwyTIXS6NC9ZZX-BxVQ1tNWjyhYwyS9MONA.

And here is one from a day ago where the indexing wasn't completed: https://viewblock.io/arweave/tx/XHXEh4UEvY8LZFyco1SkfiRlVb-oQ15LQNNQDK4DH58

For solana you can check https://explorer.solana.com/ or https://status.solana.com/. In particular, look at the Ping tx_loss (Mainnet-Beta), which should be close to zero. Been over 50% for the last hour and up and down all day. You can also try a small transfer with the cli to see if goes right through or gets hung up. If that gets hung up, the arloader with sol option will also likely get hung up.