search

IBBoard / cawbird

A fork of the Corebird GTK Twitter client that continues to work with Twitter
https://ibboard.co.uk/cawbird/
GNU General Public License v3.0
309 stars 37 forks source link

Selected File is not an image or video #272

Closed ZachBacon closed 3 years ago

ZachBacon commented 3 years ago

Describe the bug As this might be more of a gtk/gstreamer issue within an unsupported platform. But when selecting a png image, Cawbird is saying selected file is not an image.

To Reproduce Steps to reproduce the behavior:

  1. in msys2, install necessary packages pacman -S mingw-w64-x86_64-gstreamer mingw-w64-x86_64-gst-plugins-good mingw-w64-x86_64-gst-plugins-bad mingw-w64-x86_64-gst-libav mingw-w64-x86_64-vala mingw-w64-x86_64-gspell mingw-w64-x86_64-meson
  2. build cawbird using meson but with the option -Dx11=false.
  3. install cawbird and then run
  4. when composing tweet, go to attach an image, select image and see message

Expected behavior To be able to attach an image and post it successfully.

Screenshots If applicable, add screenshots to help explain your problem. https://user-images.githubusercontent.com/8948676/103808940-bd4f4400-5026-11eb-825f-aabf4250db45.mp4

System details:

Additional context Add any other context about the problem here.

Running G_MESSAGES_DEBUG=cawbird cawbird can give useful debug output. cawbird.log

Now I know this isn't really a supported platform and I'm most likely not going to receive much help in this case, but I can at least make tweets even if I can't upload images and I would honestly love to see this functionality work correctly on Windows and would try to help out where I can.

ZachBacon commented 3 years ago

Also as per https://twitter.com/CawbirdClient/status/1346895109128937476 could very much be a mimetype issue within windows.

IBBoard commented 3 years ago

Can you test that and see what it says? We should be getting an image/… or a video/… mime type, but given that Windows normally trusts file extensions then it wouldn't surprise me if it's blank or wrong!

ZachBacon commented 3 years ago

@IBBoard so did a git clone of the repo, it's showing it as png

ComposeTweetWindow.vala:529: C:\Users\zacht\Pictures\6ff32913-74f3-487c-8446-d2877b51b06a-profile_image-300x300.png has content type .png

So it's recognizing but it's just not accepting it.

ZachBacon commented 3 years ago

Also forgot to mention, images are not being shown for tweets from others.

IBBoard commented 3 years ago

Yeah, that's not a mime type, though. That's a file extension. It should be image/png. Windows is pretending it knows what to do, but it doesn't.

We can probably work around that, but it's always going to have its limits. We would basically need to list every single file extension and categorise appropriately. Which is okay for PNG (always .png) and GIF (always .gif) and WEBP (always .webp) and isn't terrible for JPEG (mostly .jpg but sometimes .jpeg). But it's a nightmare for video (.mpeg? .mpg? .mp4? .m4v? maybe .avi? what else?)

The other alternative is that msys2 fixes its GLib.FileInfo handling to return actual mime types rather than file extensions, which will solve it for every application.

Not loading images is a separate issue, and I don't know why it would occur. It shouldn't be anything to do with file types, though. But I'd normally expect errors in the log.

ZachBacon commented 3 years ago

video I can say as per twitters faq that it's

We currently support MP4 and MOV video formats on mobile apps.

On the web, we support the MP4 video format with H264 format with AAC audio. You can upload videos up to 512MB, however you will be prompted to edit videos to 2 minutes and 20 seconds or less in length.

As for the glib issue, the fileinfo stuff shouldn't be up to msys2 to patch, in the short term yes they probably could, but that would be more of an upstream issue with gnome/glib itself.

IBBoard commented 3 years ago

Yeah, I've been reading the Twitter API docs on media recently. It's… lacking detail. And there's all sorts of weird stuff with videos between codecs and containers and files and it all just gets a bit messy. Maybe AVI isn't a valid MP4 container, but I've certainly seen far more MP4 file extensions than for most files (some of it, like m4v, influenced by Apple).

Maybe the FileInfo stuff is an upstream GLib problem, but I assumed that they had done some porting of the library to Windows and so had implemented the appropriate backend. But I guess there have been other GTK things on Windows, so there might be an official one and that might be its behaviour. The Vala docs are clear that there's special behaviour for symlink checking on Windows but don't mention anything about get_content_type

ZachBacon commented 3 years ago

I'll do some digging, but the docs I'm reading are lacking in that regard.

IBBoard commented 3 years ago

Can you try the bug272-windows-cannot-do-mimetypes branch? I've put in a new config switch and some code that should work around the general shortcomings of Windows and file types.

(Users on Linux can enable it as well, but… why would you?)

ZachBacon commented 3 years ago

I'll give it a go

ZachBacon commented 3 years ago

../src/MediaUpload.vala:38.18-38.29: error: syntax error, expected =' return "image/webp"; ^^^^^^^^^^^^ ../src/MediaUpload.vala:49.5-49.32: error: properties are not allowed in(NULL)' public string media_category { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:73.5-73.25: error: properties are not allowed in (NULL)' public int64 media_id { ^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:83.5-83.25: error: properties are not allowed in(NULL)' public int64 filesize { ^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:89.5-89.26: error: properties are not allowed in (NULL)' public double progress { ^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:99.5-99.39: error: properties are not allowed in(NULL)' public GLib.Cancellable cancellable { get; private set; } ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:100.5-100.39: error: signals are not allowed in (NULL)' public signal void progress_updated(double progress); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:101.5-101.40: error: signals are not allowed in(NULL)' public signal void progress_complete(GLib.Error? error = null); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:102.5-102.40: error: signals are not allowed in (NULL)' public signal void media_id_assigned(); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ../src/MediaUpload.vala:104.12-104.22: error: syntax error, following expression/statement delimiter;' missing public MediaUpload(string filepath, bool for_dm = false) throws GLib.Error { ^^^^^^^^^^^ ../src/MediaUpload.vala:104.24-104.29: error: syntax error, following expression/statement delimiter )' missing public MediaUpload(string filepath, bool for_dm = false) throws GLib.Error { ^^^^^^ ../src/MediaUpload.vala:104.24-104.29: error: syntax error, following expression/statement delimiter;' missing public MediaUpload(string filepath, bool for_dm = false) throws GLib.Error { ^^^^^^ ../src/MediaUpload.vala:104.39-104.39: error: syntax error, expected identifier public MediaUpload(string filepath, bool for_dm = false) throws GLib.Error { ^ ../src/MediaUpload.vala:111.21-111.23: error: expected end of file cancellable = new GLib.Cancellable(); ^^^ ../src/MediaUpload.vala:104.5-102.43: warning: main blocks are experimental ../src/MediaUpload.vala:112.5-112.5: error: expected end of file } ^ ../src/MediaUpload.vala:111.21-111.19: warning: main blocks are experimental cancellable = new GLib.Cancellable();

../src/MediaUpload.vala:111.21-111.19: error: The root namespace already contains a definition for `main' cancellable = new GLib.Cancellable();

../src/MediaUpload.vala:104.5-102.43: note: previous definition of `main' was here [6/179] Generating cawbird_resources_c with a custom command ninja: build stopped: subcommand failed.'

IBBoard commented 3 years ago

Ugh, apparently I rushed that last bit and didn't test it after the last changes. All of that is because of one missing space.

And it turns out I'd missed PNGs as well in one part of the upload code.

I can't guarantee that it'll work for every video and weird variation on "normal" "supported" file extensions, but it'll be about the best that a workaround can be.

ZachBacon commented 3 years ago

I'll give it a go in a little bit, dealing with a bit of a headache.

ZachBacon commented 3 years ago

2021-01-07 Some progress was made, just won't send now at all. Same thing shows up for video as well.

IBBoard commented 3 years ago

Any logging? I successfully attached a PNG, JPG, JPEG, and non-animated GIF and then an MP4 to a draft post using this branch with the "mswindows" build flag set.

IBBoard commented 3 years ago

I can't test this on Windows and it works on Linux as an "mswindows detection approach" build. If you can provide logs of the upload problem then I can try to fix it. If you can't then I'll have to assume that it was a network issue (or some kind of Windows library issue) and close the ticket.

ZachBacon commented 3 years ago

I can't test this on Windows and it works on Linux as an "mswindows detection approach" build. If you can provide logs of the upload problem then I can try to fix it. If you can't then I'll have to assume that it was a network issue (or some kind of Windows library issue) and close the ticket.

Sorry been kinda busy, I'll try and provide as much detail as I can for this hopefully later tonight

ZachBacon commented 3 years ago

Sorry things gotten busy, I'll do a quick test tomorrow morning before I go to work.

ZachBacon commented 3 years ago

@IBBoard sorry for the delay, been literally a busy few days to say the least, on top of getting ransomwared, to which I had to reinstall windows, to restarting my dev environment, this is the raw output of cawbird and even trying to attach an image. cawbird.txt

IBBoard commented 3 years ago

That looks like a full log, which might include the content of DMs.

The important part is right at the end. It appears to call the "INIT" API endpoint and then not do anything afterwards. I'd expect either a chunk upload or an error after that. It correctly specifies the media category and media type, though, so the code is doing the right thing.

I'll look at it more over the weekend.

ZachBacon commented 3 years ago

That looks like a full log, which might include the content of DMs.

The important part is right at the end. It appears to call the "INIT" API endpoint and then not do anything afterwards. I'd expect either a chunk upload or an error after that. It correctly specifies the media category and media type, though, so the code is doing the right thing.

I'll look at it more over the weekend.

Yeah a bit does contain stuff, but nothing personal etc. I tend to not really use direct messaging anyways.

IBBoard commented 3 years ago

So, looking at the upload function (https://github.com/IBBoard/cawbird/blob/master/src/util/TweetUtils.vala#L691) it should:

I've added more debugging to the branch to see what it does… and noticed that we don't log the APPEND call because we use the upload function directly rather than Cb.Utils.load_threaded_async. So it might be something in the append that failed. It definitely wasn't getting to FINALIZE.

If you can test the branch again and just upload the section from the first /media/upload.json?…command=INIT… API call then I'll see what's happening.

ZachBacon commented 3 years ago

Just woke up, but after a quick coffee and food, I'll definitely do that.

ZachBacon commented 3 years ago

Here's the log as it's still not working @IBBoard I've also included the png file in question just in case. cawbird.txt

fly_on_screen

IBBoard commented 3 years ago

That looks like it's not hitting any of the extra debugging. Which means it never returns from the Cb.Utils.load_threaded_async call. Which is… weird. I'll have to think about how I can debug that.

It's not image related, though, because at that point we've not opened the file or sent any of its data to Twitter. We've literally just said that we've got an tweet_image file (i.e. an email for a tweet rather than a DM) that's of type image/png and it is 3472 bytes, and we've asked Twitter for an upload ID so that we can upload the content.

The only thing that's different between that and normal posting is that we've got a separate OAuth proxy for making the REST calls, because it's uploads.twitter.com rather than api.twitter.com.

IBBoard commented 3 years ago

Okay, I've added some more debugging. It'll make everything much more verbose, because the functions with the debugging are used in multiple places, but you should end up with something like this when you upload:

(cawbird:32557): cawbird-DEBUG: 19:08:27.015: ComposeTweetWindow.vala:514: Loading /home/ibboard/temp/Screenshot from 2021-01-23 18-41-34.png
(cawbird:32557): cawbird-DEBUG: 19:08:27.062: REST: POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=187455&command=INIT&media_type=image/png
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: REST: POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=187455&command=INIT&media_type=image/png finished
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: call_done_cb payload: {"media_id":1353057022216859649,"media_id_string":"1353057022216859649","expires_after_secs":86399,"media_key":"3_1353057022216859649"}
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: Parsing JSON
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: Got JSON node
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: TweetUtils.vala:708: Completed call
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: TweetUtils.vala:723: JSON was not null. Got media ID 1353057022216859649
(cawbird:32557): cawbird-DEBUG: 19:08:27.430: TweetUtils.vala:727: Got file handle
…
ZachBacon commented 3 years ago

I'll test the changes when I get home, thanks for working on this. If this begins to work, I'd love to be able to package this for not only msys2 users but also provide an installable package for general windows users if you'd allow it.

IBBoard commented 3 years ago

Cawbird is open source, so we can't stop people packaging it 😉 But if you make an msys2/Windows build then we'll definitely include it in the "community builds" list 🙂

ZachBacon commented 3 years ago

Took me a little longer, anyways updated with a git pull, using G_MESSAGES_DEBUG=cawbird cawbird &> cawlog.txt I was able to generate this output

cawbirdlog.txt

IBBoard commented 3 years ago

Wait, what? We're still seeing POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=3472&command=INIT&media_type=image/png and NOTHING else afterwards. That's not right at all.

There must be something happening with it failing to initialise connections. But also not failing. Can you try and leave it for at least a minute. Maybe five. Enough time for the connection to time out.

The one thing that's different about the upload is that it's on a separate domain. Same credentials, but an "upload.twitter.com" domain. It might be that you've got some issues accessing that domain.

Can you try ping upload.twitter.com and opening https://upload.twitter.com/ in a browser to see what happens?

ZachBacon commented 3 years ago

Yeah, I'll try when I get off work in a few hours.

On Mon., Jan. 25, 2021, 3:11 p.m. IBBoard, notifications@github.com wrote:

Wait, what? We're still seeing POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=3472&command=INIT&media_type=image/png and NOTHING else afterwards. That's not right at all.

There must be something happening with it failing to initialise connections. But also not failing. Can you try and leave it for at least a minute. Maybe five. Enough time for the connection to time out.

The one thing that's different about the upload is that it's on a separate domain. Same credentials, but an "upload.twitter.com" domain. It might be that you've got some issues accessing that domain.

Can you try ping upload.twitter.com and opening https://upload.twitter.com/ in a browser to see what happens?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IBBoard/cawbird/issues/272#issuecomment-767083197, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEIXRC6NTSYATEGM3BSPW3S3XF7RANCNFSM4VX57UGQ .

IBBoard commented 3 years ago

Any luck testing it?

ZachBacon commented 3 years ago

Sorry, been busy the last few days, I got some free time today, I'll post the results later.

ZachBacon commented 3 years ago

@IBBoard still running the app, gonna give it 20 minutes, and I got a ping on upload.twitter.com but opening up the page gives this: The Twitter REST API v1 is no longer active. Please migrate to API v1.1. https://dev.twitter.com/docs/api/1.1/overview.

ZachBacon commented 3 years ago

Alright, let it run, here's the result cawlog2.txt

One thing I'm noticing is this on line 55

"media_id_string":"1355595355530948610","expires_after_secs":86399,"media_key":"3_1355595355530948610"}

IBBoard commented 3 years ago

Yeah, the URL won't give you anything useful. But at least it loads.

(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: REST: POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=3472&command=INIT&media_type=image/png finished
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: call_done_cb payload: {"media_id":1355595355530948610,"media_id_string":"1355595355530948610","expires_after_secs":86399,"media_key":"3_1355595355530948610"}
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: Parsing JSON
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: Got JSON node
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: TweetUtils.vala:708: Completed call
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: TweetUtils.vala:723: JSON was not null. Got media ID 1355595355530948610
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: TweetUtils.vala:727: Got file handle
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.810: TweetUtils.vala:743: Reading bytes for chunk 0…
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.811: TweetUtils.vala:745: Read chunk of 3472 bytes
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.811: TweetUtils.vala:764: Calling upload
(cawbird.exe:9400): cawbird-DEBUG: 14:14:52.811: TweetUtils.vala:766: Awaiting upload completion

So, it does the INIT and then does the upload. And it's (obviously) only a single chunk. But it never seems to get out of the upload loop.

That code is asynchronous, though, so I've added what's hopefully one last bit of logging. We might be seeing a race condition between the upload completing and the bit where we say "wait until the upload completes".

IBBoard commented 3 years ago

Any luck with 4daa0fa?

Unless I've got enough info to work out why it's failing for you and not me then I'm probably going to merge up to 04339a2 (before I put in the debugging)

ZachBacon commented 3 years ago

Sorry, I haven't been able to do much testing yet, recently got laid off my job and searching for a new one, I'll do an updated test tonight and send the results tomorrow.

On Fri., Feb. 12, 2021, 3:50 p.m. IBBoard, notifications@github.com wrote:

Any luck with 4daa0fa https://github.com/IBBoard/cawbird/commit/4daa0fa9759d8bb57fc597d41072abb43963057a ?

Unless I've got enough info to work out why it's failing for you and not me then I'm probably going to merge up to 04339a2 https://github.com/IBBoard/cawbird/commit/04339a2ee11c5928ae26b3295502cada2411b5ed (before I put in the debugging)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IBBoard/cawbird/issues/272#issuecomment-778447698, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEIXRF34IU4I4BM7UX22MLS6WIC7ANCNFSM4VX57UGQ .

IBBoard commented 3 years ago

Sorry to hear that. Don't worry - life comes first.

ZachBacon commented 3 years ago

Sorry to hear that. Don't worry - life comes first.

Got some free time today, I'll get my local copy updated and see if I can pull some more logs for you, will try and get as much as I can and post later today

ZachBacon commented 3 years ago

@IBBoard Got the log, even tried to attach a few images as a test, hopefully it will give you something useful. cawbird-log.txt

IBBoard commented 3 years ago

That… doesn't make sense. I can see it hit the INIT call (line 710) but it never completes the yield (logging on line 711) or errors (error handling with warning on 714-717). It just hangs.

But previously you were seeing it at least try the append, because it got to "Awaiting upload completion"

I'll have to look into the length of the timeout. It looks like you left them from 06:11:24 to 06:11:28 (four seconds), from 06:11:33 to 06:13:20 (nearly two minutes) and from 06:13:28 to 06:13:32 (four seconds). So, if it is hanging for some unknown reason then it takes at least two minutes to time out (if at all). It won't be a solution, but it might at least let us confirm that the connection is hanging for some reason.

IBBoard commented 3 years ago

Apparently it should be timing out after 60 seconds.

We use librest. Librest creates a SoupSession object. The default timeout is 60s for SoupSession or infinite for Sync/ASync. But they're specific subtypes with their own constructors.

Which means that even if it doesn't work properly we should have seen one of: returns to line 711, errors with a specific problem to line 714, or errors with a timeout (also to line 714).

I think I need to flood the logs a bit more!

IBBoard commented 3 years ago

Looks like I'd already put in most of the logging that I was thinking I needed to add 😄

Here's the output that I get when uploading an image with the latest commit, built with meson configure -Dmswindows=true build (effectively falling back to "trust the file extension", even though Linux handles mime types properly):

(cawbird:28999): cawbird-DEBUG: 19:59:02.017: ComposeTweetWindow.vala:514: Loading /home/ibboard/temp/tK143AK.jpg
(cawbird:28999): cawbird-DEBUG: 19:59:02.071: Cancellable? not cancelled
(cawbird:28999): cawbird-DEBUG: 19:59:02.071: REST: POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=542115&command=INIT&media_type=image/jpeg
(cawbird:28999): cawbird-DEBUG: 19:59:02.071: cb_utils_load_threaded_async fired
(cawbird:28999): cawbird-DEBUG: 19:59:02.071: TweetUtils.vala:722: Invoked
(cawbird:28999): cawbird-DEBUG: 19:59:02.371: call_done_cb
(cawbird:28999): cawbird-DEBUG: 19:59:02.371: REST: POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=542115&command=INIT&media_type=image/jpeg finished
(cawbird:28999): cawbird-DEBUG: 19:59:02.371: call_done_cb payload: {"media_id":1362491836296790016,"media_id_string":"1362491836296790016","expires_after_secs":86399,"media_key":"3_1362491836296790016"}
(cawbird:28999): cawbird-DEBUG: 19:59:02.371: Parsing JSON
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: Got JSON node
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:711: Call returned
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:714: Returned root
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:724: Completed call
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:739: JSON was not null. Got media ID 1362491836296790016
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:743: Got file handle
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:759: Reading bytes for chunk 0…
(cawbird:28999): cawbird-DEBUG: 19:59:02.372: TweetUtils.vala:761: Read chunk of 542115 bytes
(cawbird:28999): cawbird-DEBUG: 19:59:02.373: TweetUtils.vala:780: Calling upload
(cawbird:28999): cawbird-DEBUG: 19:59:02.373: TweetUtils.vala:782: Awaiting upload completion
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 119 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 138 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 258 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 264 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 390 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 391 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 587 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 16971 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 33355 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.431: TweetUtils.vala:682: Upload progress: 49739 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.432: TweetUtils.vala:682: Upload progress: 66123 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.432: TweetUtils.vala:682: Upload progress: 82507 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.432: TweetUtils.vala:682: Upload progress: 98891 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.466: TweetUtils.vala:682: Upload progress: 115275 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.466: TweetUtils.vala:682: Upload progress: 131659 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.466: TweetUtils.vala:682: Upload progress: 148043 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.466: TweetUtils.vala:682: Upload progress: 164427 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.466: TweetUtils.vala:682: Upload progress: 180811 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.477: TweetUtils.vala:682: Upload progress: 197195 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 213579 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 229963 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 246347 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 262731 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 279115 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 295499 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 311883 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.478: TweetUtils.vala:682: Upload progress: 328267 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.539: TweetUtils.vala:682: Upload progress: 344651 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.539: TweetUtils.vala:682: Upload progress: 361035 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.539: TweetUtils.vala:682: Upload progress: 377419 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.540: TweetUtils.vala:682: Upload progress: 393803 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.540: TweetUtils.vala:682: Upload progress: 410187 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.540: TweetUtils.vala:682: Upload progress: 426571 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.540: TweetUtils.vala:682: Upload progress: 442955 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.540: TweetUtils.vala:682: Upload progress: 459339 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 475723 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 492107 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 508491 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 524875 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 541259 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.586: TweetUtils.vala:682: Upload progress: 542702 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: TweetUtils.vala:682: Upload progress: 542774 of 542774
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: TweetUtils.vala:687: Finished uploading chunk
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: TweetUtils.vala:784: Chunk uploaded
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: TweetUtils.vala:798: Finalising upload
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: Cancellable? not cancelled
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: REST: POST 1.1/media/upload.json?command=FINALIZE&media_id=1362491836296790016
(cawbird:28999): cawbird-DEBUG: 19:59:02.881: cb_utils_load_threaded_async fired
(cawbird:28999): cawbird-DEBUG: 19:59:03.635: call_done_cb
(cawbird:28999): cawbird-DEBUG: 19:59:03.635: REST: POST 1.1/media/upload.json?command=FINALIZE&media_id=1362491836296790016 finished
(cawbird:28999): cawbird-DEBUG: 19:59:03.635: call_done_cb payload: {"media_id":1362491836296790016,"media_id_string":"1362491836296790016","media_key":"3_1362491836296790016","size":511923,"expires_after_secs":86400,"image":{"image_type":"image\/jpeg","w":1614,"h":1872}}
(cawbird:28999): cawbird-DEBUG: 19:59:03.635: Parsing JSON
(cawbird:28999): cawbird-DEBUG: 19:59:03.636: Got JSON node
(cawbird:28999): cawbird-DEBUG: 19:59:03.636: MediaUpload.vala:122: Finalizing upload
ZachBacon commented 3 years ago

But your build is done via native Linux right?

IBBoard commented 3 years ago

Yeah, built and running on Linux, but using the "Windows doesn't understand mimetypes" code path. There's not actually anything in the code that's Windows specific, even with these changes.

What was basically happening before was that we weren't generating the POST 1.1/media/upload.json?media_category=tweet_image&total_bytes=542115&command=INIT&media_type=image/jpeg request correctly because we ended up with …&media_type=.jpg (we actually failed a little before that because of internal checks, but that's where we would have got to). As long as we can build those requests correctly (which your logging shows that we do) then it should be identical on Windows and Linux.

ZachBacon commented 3 years ago

Could be an issue related to librest or libsoup being used on windows. Which means I'll have have to investigate myself, whether it's their issue tracker or trying to write tests to check if it's those libraries or not.

IBBoard commented 3 years ago

Possibly. But all of the other calls use librest/libsoup (fetching tweets, posting tweets, etc). And your latest log showed it starting but never completing the INIT step, which is just plain JSON. We don't add the data until the APPEND step. And even then it should be encoded as text, so it shouldn't look that different to a JSON request.

ZachBacon commented 3 years ago

This may not be crazy, but I have a feeling it could be Windows firewall, I have a sinking suspicion that it's Windows firewall. I'm going to try something locally and see if my suspicions are correct.

IBBoard commented 3 years ago

Good old security if it is 😄

We still need the "don't trust the mime type" code, but it might explain the hung connections when everything else works fine (because it's sending stuff to a slightly different Twitter domain for uploads)