How would you display the progress if we used the way you mentioned?
Something like this 1.56/2.85 MB
Or maybe something else?
The problem when GitHub is creating a zip is that we don't know how large the full file will be. Even GitHub doesn't know. So it sends Content-Length: 0 and then starts creating a zip/tar which gets directly piped into the web server output.
So even a web browser that downloads those archives from GitHub just shows how much it's currently received, such as "102.1 MiB", "180.6 MiB", etc etc, with a basic spinner, without any progress bar (since it doesn't know), until it's finally done.
In that scenario, we only know one value: The current bytes downloaded.
Therefore the only thing we can do for the user is to have two types of progress displays:
progress = 15, is_percentage = true: In this (normal mode) the progress says "15%".
progress = 137123756, is_percentage = false: In this (bytes mode), the progress says covert_bytes_to_string(progress) to generate a human-readable byte value such as "1.5 MiB", "1.9 MiB" etc, so that the user at least sees that the download is progressing. (And yeah, covert_bytes_to_string is a typo in utils/filesystem.vala. :D)
And is_percentage = download_size > 0 is the way we determine that. Basically if the server has told us the filesize, we can use a percentage. If it didn't say the size, we need to use bytes.
This would indeed be a robust improvement which is unrelated to the STL implementation, so I created this ticket to track that.
The main thing to be able to implement this is to change the Download handler's progress_callback to take 2 parameters (int64 progress and bool is_percentage). And yeah I think we should fetch the downloaded bytes (and calculated progress) as a 64-bit integer just to be safe, since 32-bit ints can only fit file sizes up to \~4 GiB. (Which is precisely why the human-readable size converter is already defined as covert_bytes_to_string (int64 size). :))
Then, the progress_callback is responsible for rendering as a % if it received a percentage value, otherwise convert it to bytes and display as human-readable file size (rendered in the same text-label as the percentage would use).
Edit: I haven't done a full review of src/utils/web.vala, but a quick glance seems like it's almost 64-bit aware.
It already uses var server_download_size = soup_message.get_response_headers ().get_content_length (); which should already automatically be 64-bit (I can't imagine that they'd use anything less). Edit: Yes it's defined as returning int64: https://valadoc.org/libsoup-3.0/Soup.MessageHeaders.get_content_length.html
The "downloaded bytes counter" is ulong bytes_downloaded = 0; which should be changed to int64 bytes_downloaded = 0; for extra clarity.
And don't worry about uint vs int. There's no way we will ever download a file that overflows a 64-bit signed integer. :) That's like petabytes of storage. So we use int64 since that lets us use negative numbers for status codes, and it matches the soup get_content_length value type.
I just noticed that there was a question in one of the PRs and I hadn't seen it before.
The problem when GitHub is creating a zip is that we don't know how large the full file will be. Even GitHub doesn't know. So it sends
Content-Length: 0
and then starts creating a zip/tar which gets directly piped into the web server output.So even a web browser that downloads those archives from GitHub just shows how much it's currently received, such as "102.1 MiB", "180.6 MiB", etc etc, with a basic spinner, without any progress bar (since it doesn't know), until it's finally done.
In that scenario, we only know one value: The current bytes downloaded.
Therefore the only thing we can do for the user is to have two types of progress displays:
progress = 15, is_percentage = true
: In this (normal mode) the progress says "15%".progress = 137123756, is_percentage = false
: In this (bytes mode), the progress sayscovert_bytes_to_string(progress)
to generate a human-readable byte value such as "1.5 MiB", "1.9 MiB" etc, so that the user at least sees that the download is progressing. (And yeah,covert_bytes_to_string
is a typo inutils/filesystem.vala
. :D)is_percentage = download_size > 0
is the way we determine that. Basically if the server has told us the filesize, we can use a percentage. If it didn't say the size, we need to use bytes.This would indeed be a robust improvement which is unrelated to the STL implementation, so I created this ticket to track that.
The main thing to be able to implement this is to change the Download handler's
progress_callback
to take 2 parameters (int64 progress
andbool is_percentage
). And yeah I think we should fetch the downloaded bytes (and calculated progress) as a 64-bit integer just to be safe, since 32-bit ints can only fit file sizes up to \~4 GiB. (Which is precisely why the human-readable size converter is already defined ascovert_bytes_to_string (int64 size)
. :))Then, the
progress_callback
is responsible for rendering as a%
if it received a percentage value, otherwise convert it to bytes and display as human-readable file size (rendered in the same text-label as the percentage would use).Edit: I haven't done a full review of
src/utils/web.vala
, but a quick glance seems like it's almost 64-bit aware.var server_download_size = soup_message.get_response_headers ().get_content_length ();
which should already automatically be 64-bit (I can't imagine that they'd use anything less). Edit: Yes it's defined as returningint64
: https://valadoc.org/libsoup-3.0/Soup.MessageHeaders.get_content_length.htmlulong bytes_downloaded = 0;
which should be changed toint64 bytes_downloaded = 0;
for extra clarity.int64
since that lets us use negative numbers for status codes, and it matches the soupget_content_length
value type.