Upload tag information, processed < total

[ldeffenb]

Summary

When uploading a certain file to the swarm with the /files API with a swarm-tag specified, the tag never gets finished.

Steps to reproduce

Create a new tag UID: curl -X POST http://192.168.10.17:1633/tags

Upload the offending file (source below): curl --data-binary @t(134)-p(133)-s(133)-11-0-0-18-76-128.meta -H "swarm-tag: 1727033384" http://192.168.10.17:1633/files?name=t(134)-p(133)-s(133)-11-0-0-18-76-128.meta

Monitor the tag: curl http://192.168.10.17:1633/tags/1727033384 {"uid":1727033384,"startedAt":"2021-02-11T14:43:44.607803218-05:00","total":134,"processed":133,"synced":133}

That was 25 minutes ago and still processed has not gotten to total. And in my experience with other, similar files, it never will.

Check the debug version of the tag info: curl http://192.168.10.17:1635/tags/1727033384 {"total":134,"split":133,"seen":9,"stored":133,"sent":125,"synced":124,"uid":1727033384,"address":"","startedAt":"2021-02-11T14:43:44.607803218-05:00"}

Yes, it shows that some chunks were previously seen, but even so, seen+stored is still only 133 which matches split, but has not arrived at total.

You can get the file from Bee CID:361d463da6e69040c98b09267d18613e47f742ec47fea1d98d3a5fc201fc93b3

or if that doesn't work, IPFS should have the original at: https://ipfs.io/ipns/12D3KooWDUayfDzwEv6DhAqHc8hQBKZXFEzXk5wA1V4wu63kke8P/11/0/0/18/76/128.meta

or without IPNS at: https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/18/76/128.meta

another instance with the same 133/134: https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/67/145/136.meta https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/34/60/0.meta https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/34/142/136.meta https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/68/153/136.meta

I'm seeing a commonality here. The file sizes (in bytes) of those 3 are: 128.meta: 525,882 136.meta: 527,396 0.meta: 527,550 136.meta(2): 526,820 136.meta(3): 526,952

As was predicted by @zbiljic , here are a few files that hang at 262/263 processed vs total: https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/66/172/0.meta 1,048,055 bytes https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/66/141/8.meta 1,049,230 bytes https://ipfs.io/ipfs/QmQukdK4D5dt8eEkFhiHf5SCW5bvh4XLxfSWEJtZT8Ksik/11/0/0/18/200/8.meta 1,048,908 bytes

Expected behavior

I expected processed to get to total (normal API) and/or split to get to total (debug API).

Actual behavior

processed and split end up one short of total and stay there.

This makes it really difficult to wait for an upload to be completely processed. I do get the reference response back from the files API, but I would still expect the tag monitor to finish as well.

[ldeffenb]

BTW, I did the same file to two different nodes, both on 0.5.0, and the hang situation was the same. 133/134.

[zbiljic]

@ldeffenb It would seem you have encountered some edge-cases in the function for estimating total number of chunks. Apart from "simplifying" a bit public API tag response, that change also added estimation for total number of chunks based on the size being uploaded.

I feel that here I need to explain that there are two distinct behaviors for tags. One can create a tag beforehand, and then use if by specifying swarm-tag header in one (or more) upload request(s). Other is when one uploads some content and receives response with reference for the uploaded content, where swarm-tag header will be returned with tag which was automatically generated for that upload. The tag in the second case is basically only useful to track if the underlying content is properly synced and distributed though the network. While for the first case, same tag can be used for multiple uploads (if one wishes to do so), and track progress for everything.

One main difference between these two paths is that in the second (tag UID after upload), the tag will be marked as "done" internally, which will then copy internal split value (correct number of chunks) into total, while in the first case the tag is not done (because it was created before), and therefore it will just contain that estimated number of chunks for the upload.

As can be seen from my comment, this is function only "estimates" total number of chunks that will be produced; it makes no guarantees about correctness. One thing that it should provide (however, again it does not guarantee), is that estimated total value will be greater or equal that processed and synced count. This should allow for good enough progress monitoring.

Once you are done with tag, and do not plan to use it for other uploads, my suggestion is to mark tag as done, by performing PATCH /tags/{uid} request. This will perform same process as I noted in the second case here, which should update tag to have correct total value.

As far as this specific issue, there seems to be problem estimator function with sizes that are just above 524288 bytes (and less than 528385 bytes) in size. This means for all content whose size when converted to chunks comes at 128 chunks, i.e. 525882 / 4096 = 128.38xxx. The 128 is equal to internal branching factor (for non-encrypted addresses), at which point additional chunk is created, etc. not go into implementation. It might be possible that we have the same problem with sizes that are multiple of this branching factor (by branching factor).

I will see about scheduling this, but the main point is that for manually created tags value in total is only estimation until tag is manually marked as done.

[ldeffenb]

@zbiljic I understand everything you said, and thank you for taking the time to provide the detail. The fact that total is only an estimation is certainly not mentioned anywhere that I've noticed in the documentation. I am pre-creating the tag and specifying it as the swarm-tag on the upload request. By taking this approach, I can monitor the splitting and syncing progress, while option 2 (getting the swarm-tag from the upload response) doesn't give me a chance to monitor the split because it is complete before the CID can be issued in the upload response.

But as far as "good enough progress monitoring", I would have to partially disagree. You are correct as far as "monitoring" goes, but with the details documented above, it is impossible to get to 100% when the processed never quite gets to total, and therefore impossible to know that the splitting is complete vs just stalled for some reason.

I will consider the PATCH API, and might issue that once the upload response is received to both attach the final CID to the tag and benefit from the adjustment of total which would then, presumably, allow the asynchronous progress monitor to finally see processed get to the (now reduced) total value.

But unless you find a way to emphatically document that the Total is only an estimate and that processed cannot be trusted to ever get to that value (without external PATCH intervention), then IMHO the estimator bug really needs to be fixed and have the edge case(s) removed.

As a data point, so far I've only seen the 133/134 happen and I've had files that were more than 266/268, but possibly not the exact size to see it happen at the next multiple. I've updated my uploader to record which files take longer than 3 minutes to "finish" (processed >= total) along with their tags, so I can go back and look at it later. I'll let you know if I get any that are not in the 133/134 chunk range.

At least you have multiple file samples to test the correction to the estimator!

[ldeffenb]

Just a crazy thought. I haven't looked at the upload code, but I can imagine something like:

if swarm-tag is specified, generate an estimate and add it to Total on that tag else create a new tag to track the upload. Do the actual upload, accumulating the actual split value somewhere? When upload is complete, if a new tag was created, correct total based on the actual processed/split value.

Could the first step be updated to remember the estimated total in addition to adding it to the client-specified tag and the final step be updated to compare the estimated total to the actual processed/split value and adjust the client-specified tag's total by the difference between estimate and actual. This would be at the same point that a newly-created upload-tracking tag would be automatically fixed, but would apply to client-specified, pre-existing tags.

If so, then the client-specified tag will be corrected once the actual total is known and the estimator won't have to a) get fixed to be correct now, and b) keep up with changes in the actual splitting layer that may render the corrected estimator incorrect again at some point in the future.

[zbiljic]

Something like that could be possible if we can say that tag will ever be used for single upload.

As soon as we consider that tag can be used for multiple uploads, which may occur simultaneously, we lose ability to know split value for each separate upload. In that case, we cannot check difference between split and total values, to correct the total that was estimated.

[ldeffenb]

So the individual upload doesn't really know it's own split count, just whatever has been accumulated into the tracking tag, so only a locally-created one can be used. I get that. But would it be possible, again not having looked at the source, to get the actual split count back into the upload "thread" via a route other than the tracking tag? That way both the original estimated total and the actual single-upload split could be known for the final total adjustment. Or is the process so asynchronous that the actual split count cannot be "returned" to the upload except via the tracking tag increments?

[zbiljic]

Current code is not configured to track for separate uploads. I am not saying that it would be impossible to somehow support requested feature, but it would introduce a lot of additional code just for this. Considering that there are ideas to change how tags work internally, I do not know if it is worth to add additional code just for this now. @Eknir ?

[ldeffenb]

Actually if the PATCH /tags/{uid} can manage to get the Total corrected given only a CID, one would think that the upload process that generated that CID would be able to get the final total and use it to delta from the original estimate! And it would only need to look this up if an original estimate was added into an existing client-specified tag to begin with.

[zbiljic]

The request to PATCH /tags/{uid} is entirely comes down to the user. There is no guarantee it will set "correct" value. It is up to the user to know that one or all uploads that used same tag are completed (and returned successful response). Only after that point is it safe to make that PATCH request.

Should the user invoke that request during active upload (before receiving reference for the upload) the tag will be marked as done, and the current state of split value be copied into total, however after that point the split and other fields may continue to increase, while the total remains with snapshot value from the time of PATCH request.

P.S. There is also some other details around current implementation, where if the tag is marked as done it will be saved in local state store (on disk), and its state will survive restarts. In that situation, one may mark tag as done (for example total will have value of 10), and the user may then use tag again, which will cause the the total and other fields to increase, all while querying that tag UID it will return "expected" values (for example the total will then be 23). If then the node is restarted, checking the state of that specific tag will return values the tag had at the time the tag was marked as done (in example noted the total will again be 10). As I noted, there may be some questionable details around tags, however there are plans to address them.

One could look at /tags API as exposed manual transaction system. User can "start" transaction (create a tag), use that transaction (tag) across one or multiple requests (uploads), and then it is up to the user to "end" transaction, but user is responsible to make sure all requests within that transaction are "completed" successfully. I know I might be reaching too much here, and knowingly ignoring some aspects in the comparison, but it is close enough.

Again, I am not saying that it would be impossible to have (as close to) "perfect" state in tags. However decisions have been made, in past or the present, related to either having clean separation of responsibilities in the code or space/time constrains. As the time goes, certain areas will be revisited, and some parts upgraded as more information about usage patterns or possible are reported.

[ldeffenb]

Ok, so the use case would be a user uploading a large file and simply wanting to monitor the chunking process and to know, from the monitor, that it is complete. This would mean:

a) Create the tag b) Start the upload with swarm-tag header c) Monitor the tag with /tags/{uid}

The current "feature" (euphemism for "bug") means that such a use must add: d) When the upload completes and the CID is returned, do a PATCH so that the asynchronous tag monitor will eventually get Processed == Total and possibly even longer get Synced==Total.

What this exposes is a recovery dilemma where the client cannot continue to monitor an existing tag if, for some connectivity or other reason, the upload data was pushed, but the CID response was not received. The monitor, in these boundary cases, would go on forever waiting for the only definitive end condition (Processed==Total). The only recovery in this situation is to create a new tag and completely upload the file again, and hope the CID is actually received this time 'round. This is quite user-abusive if the file sizes are huge.

If this bug is fixed, then the following sequence would actually work (semi-)reliably.

1) Create a tag recording the {uid} somewhere 2) Initiate the upload specifying swarm-tag 3) Monitor the tag for completion.

And if everything drops after the data has transferred, but before completion, the client can come back to step 3 to determine if the upload completed Processing and/or syncing. Again, with the current bug, some files would end up triggering a completely unnecessary re-upload sequence.

But I do understand resource constraints and prioritizing decisions. I just hope that this gets fixed somehow better than "it just works this way, live with it" before too many clients have implemented work-arounds. Just please remember that not everyone will be using your CLI or JS interfaces where you can hide such issues until (hopefully) final resolution, but may be implementing using the http API (as I am, from lua) where all the dirty laundry becomes visible.

[zbiljic]

If the reference is not received from request that was uploading how can one be entirely sure all the data is pushed and processed by the node.

As for the other comment, I think that I already phrased it before. This feature was implemented at some point according to ideas and knowledge that was available at that time. Maybe some edge cases were not covered. Maybe use case was not optimal for some things. Recent updates were made in response to some user feedback. This also is valid user feedback. The work on fixing this total estimation bug was started before my first comment here. It should be fixed at some point, or if that is not possible some other more "heavy" solution implemented around it. There are limited resources for all issues, and tasks are selected based on various decisions at the time. As mentioned, even before this issue plans were discussed for much details for tags.

Until then I think that steps for using tags are valid workaround for your use-case. I do not say that "you're using it wrong", my belief is that however one use it should be valid. However it should be noted that even if this current bug did not exist some cases could be reached which will seem as invalid.

[Eknir]

Closed in favor of #1371. Thanks for reporting and for thinking with us on how to solve this -- we'll surely come back to what is written here.