search

dvingerh / PyInstaLive

Python script to download Instagram livestreams.
MIT License
549 stars 111 forks source link

Downloaded segments not merging into a video. "source_17853001438500095_0_mp4.tmp: No such file or directory" #69

Closed CrypticSignal closed 5 years ago

CrypticSignal commented 5 years ago

Fill in this template completely. Issues not following this template will be closed and ignored.

Check the boxes below by filling [ ] with an x so it looks like [x].

Use the Preview button to ensure the template is filled in correctly.

I am using:

To report a bug, fill in the information below.

PyInstaLive information
[I] PyInstaLive version:        3.1.1
[I] Python version:             3.7.3
[I] FFmpeg framework:           Available
[I] Cookie files:               1 ([removed].json matches config user)
[I] CLI supports color:         No
[I] Command to run at start:    None
[I] Command to run at finish:   None
[I] Config file contents:

    [pyinstalive]
    username = [removed]
    password = [removed]
    download_path = C:\Users\H\Desktop\Insta
    download_lives = True
    download_replays = True
    download_comments = true
    show_cookie_expiry = True
    log_to_file = True
    ffmpeg_path =
    run_at_start =
    run_at_finish =
    use_locks = True
    clear_temp_files = False
    do_heartbeat = False
    proxy =
    verbose = False

[I] End of PyInstaLive information screen.
Command used

pyinstalive -u [username] -p [password] -d "[their username]"

Behavior
[I] Waiting for comment downloader to finish.
[I] Successfully saved 45 comments.
---------------------------------------------------------------------------
[I] Merging downloaded files into video.
C:\Users\H\Desktop\Insta/20190728_s[username]_17853001438500095_1564346301_live_downloads\source_17853001438500095_0_mp4.tmp: No such file or directory
[I] Successfully merged downloaded files into video.

It's looking for "source_17853001438500095_0_mp4.tmp" in the folder where the segments get saved and cannot find it because it does not exist. What is this anyway?

CrypticSignal commented 5 years ago

Someone just went live and I decided to start the download and then cancel it so I can show you the error I get. This is what I get:

[I] Waiting for comment downloader to finish.
[I] Successfully saved 45 comments.
---------------------------------------------------------------------------
[I] Merging downloaded files into video.
C:\Users\H\Desktop\Insta/20190728_s[username]_17853001438500095_1564346301_live_downloads\source_17853001438500095_0_mp4.tmp: No such file or directory
[I] Successfully merged downloaded files into video.

It's looking for "source_17853001438500095_0_mp4.tmp" in the folder where the segments get saved and cannot find it because it does not exist. What is this anyway?

dvingerh commented 5 years ago

EDIT: Your PyInstaLive log information shows you're not running the latest version. I'll be closing this issue until you have upgraded to the latest version and are still experiencing the problem.

I also suspect you're running the Windows executable. It would be better not to use those builds since I don't test them, they're only added upon request.

So this started happening out of the blue? It can be some kind of permissions issue. Either way, I can't reproduce the error on multiple systems. I have not made any changes that could influence the downloading/merging process either. Try a different download location.

CrypticSignal commented 5 years ago

EDIT: Your PyInstaLive log information shows you're not running the latest version. I'll be closing this issue until you have upgraded to the latest version and are still experiencing the problem.

I also suspect you're running the Windows executable. It would be better not to use those builds since I don't test them, they're only added upon request.

So this started happening out of the blue? It can be some kind of permissions issue. Either way, I can't reproduce the error on multiple systems. I have not made any changes that could influence the downloading/merging process either. Try a different download location.

Windows executable? I don't see any .exe's in your repo. And I'm not sure why it says version 3.1.1 as I downloaded the latest zip by clicking on "Clone or download". That's what I'm using.

I'll try re-downloading the zip though. I'll also try changing the download location. It's a weird bug as it's downloaded the segments in the correct download location, it's just unable to merge them into a video.

What is the significance of the .tmp file that it is looking for?

dvingerh commented 5 years ago

Executables are available for some releases at https://github.com/notcammy/PyInstaLive/releases Make sure all other possibly existing instances of PyInstaLive are uninstalled/deleted from (other) Python 2 / 3 as well so you're not accidentally running one of those instead.

The .tmp files are the download segments merged into one single file by ffmpeg. But I suppose ffmpeg fails to merge the two (mp4.tmp = video, m4a.tmp = audio) together at the end for reasons I can't determine. The final result of those two tmp files merged together again is the livestream video file.

CrypticSignal commented 5 years ago

There's obviously a piece of code that merges the segments together. Is it FFmpeg code? I have used FFmpeg on multiple occasions for other things, so if you could show me which code merges the segments together, I can try running it separately and see what error FFmpeg gives.

I've never "installed" PyInstaLive as pip install git+https://github.com/notcammy/PyInstaLive.git@3.1.7 --process-dependency-links doesn't work to me, I get this: https://i.imgur.com/pIJHVuV.png

So it's not possible for me to be running a previous installation. I always use the zip file which I extract, then I run command prompt in that directory and do pyinstalive -d "username"

dvingerh commented 5 years ago

The merging is handled by the Instagram API dependency, but you can find its code here: https://github.com/ping/instagram_private_api_extensions/blob/master/instagram_private_api_extensions/live.py#L469

PyInstaLive's assemble command works in a similar fashion: https://github.com/notcammy/PyInstaLive/blob/master/pyinstalive/assembler.py

I've never "installed" PyInstaLive as pip install git+https://github.com/notcammy/PyInstaLive.git@3.1.7 --process-dependency-links doesn't work to me, I get this: https://i.imgur.com/pIJHVuV.png

Newer versions of pip have removed --process-dependency-links. See https://github.com/notcammy/PyInstaLive/issues/35#issuecomment-462057455 for more info.

CrypticSignal commented 5 years ago

Is there another way I can get the latest version of PyinstaLive? Considering I cannot use pip and overwriting the contents of the old zip folder with the new zip folder doesn't work.

dvingerh commented 5 years ago

You can use pip by downgrading it. Use where pyinstalive to determine if there are any binaries in your PATH which can interfere as well.

If you use the zip then delete the old folder, get the newest version and run it inside that folder with python3 -m pyinstalive. Make sure to copy your configuration file to the new folder as well.

CrypticSignal commented 5 years ago

Okay I've deleted pyinstalive.exe after using finding its location with where pyinstalive. Then I copied the contents of the new zip folder in the old folder and tried running pyinstalive but got a "no option skip_merge" error.

I'm assuming I need to set that to false so I have added skip_merge = False in pyinstalive.ini

CrypticSignal commented 5 years ago

Someone started streaming and I was able to test. I'm still getting the source_18087976228030568_0_mp4.tmp: No such file or directory error and I'm on the latest version of PyInstaLive now.

CrypticSignal commented 5 years ago

If I check the folder where the downloaded segments are, I can only see .m4a and .m4v files. There is no .tmp file. The problem seems to be that it's not making a .tmp file for some reason?

dvingerh commented 5 years ago

Considering I can't reproduce the issue whatsoever there's not much I can do. Does manually assembling the segments (--assemble) work? I'm guessing Windows or ffmpeg just doesn't like the paths being given for the segments or output file.

CrypticSignal commented 5 years ago

Considering I can't reproduce the issue whatsoever there's not much I can do. Does manually assembling the segments (--assemble) work? I'm guessing Windows or ffmpeg just doesn't like the paths being given for the segments or output file.

How do I do this? Open a command prompt in the directory that the segments are in and enter python -m pyinstalive --assemble?

dvingerh commented 5 years ago

Yeah, for instance python3 -m pyinstalive --assemble 20190729_ifollowcoolpeople123_18089261065032637_1564409116_live_downloads. Ideally you pass the json file but if one is not available it'll fall back to the segment directory itself.

CrypticSignal commented 5 years ago

[E] The segment directory does not exist or does not contain any files: 20190730_username_18087976228030568_1564505807_live_downloads

dvingerh commented 5 years ago

You need to pass the absolute path since your segments folder isn't located in the same directory as PyInstaLive. For example: C:\Users\User\Documents\Git\PyInstaLive\downloaded\20190729_username_18089261065032637_1564409116_live_downloads

CrypticSignal commented 5 years ago

I get this error:

[I] Assembling video segment files from specified folder: 20190728_username_18089407207054065_1564344606_live_downloads
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-9877.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-10878.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-11879.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-12880.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-13881.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-14882.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-15883.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-16884.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-17885.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-18886.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-19887.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-20888.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-21889.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-22890.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-23891.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-24892.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-25893.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-26894.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-27895.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-28896.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-29897.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-30898.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-31899.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-32900.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-33901.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-34902.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-35903.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-36904.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-37905.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-38906.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-39907.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-40908.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-41909.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-42910.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-43911.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-44912.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-45913.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-46914.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-47915.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-48916.m4a?ms=v1_ll_hd3_t
[W] Audio segment not found: 20190728_username_18089407207054065_1564344606_live_downloads\18089407207054065-49917.m4a?ms=v1_ll_hd3_t
dvingerh commented 5 years ago

I guess we found the problem now. The external API dependency isn't properly sanitizing the segment download urls. ?ms=v1_ll_hd3_t shouldn't be in the segment filenames. This is also why it fails to generate a video file.

I suggest using some kind of bulk rename tool such as https://www.den4b.com/products/renamer to manually remove all the old url parameters from the filename and try to manually assemble again (remove any existing .tmp files) Segments should be named similar to 18089407207054065-49917.m4x with either m4a or m4v extension.

If you use the JSON file, either just delete it or remove the ?ms=v1_ll_hd3_t strings from the filenames in there as well.

CrypticSignal commented 5 years ago

I'm not sure why the errors are showing that ?ms=v1_ll_hd3_t is in the filenames because that is not in any of the filenames. The filenames are already in the 18089407207054065-49917.m4x format.

CrypticSignal commented 5 years ago

With that being said, that string is in the json file though, as you can see here: https://i.imgur.com/DjkH4wC.png

I'll remove that string from the json file and report back.

dvingerh commented 5 years ago

Ok then the actual download went fine but the dependency still messed up on sanitizing the filenames when saving them to the JSON file. Simplest way is just deleting the json file at this point, not using it hasn't caused any problems in my past tests.

CrypticSignal commented 5 years ago

After removing that string from the json file, the --assemble command has worked. It's a little annoying that every time I will need to delete the JSON file and then use the -assemble command, unless we can figure out why the filenames are not being sanitized and find a solution to that.

dvingerh commented 5 years ago

It's probably something the maintainer of https://github.com/ping/instagram_private_api_extensions has to fix as the segments are read from an array in his code rather than a file I created. Unfortunately he's not very active.

Assuming this only recently started happening(?) and doesn't occur for everyone we can only hope the problem goes away on its own now. A similar issue has been posted here before (https://github.com/notcammy/PyInstaLive/issues/47) but it didn't occur to me until now.

With the problem identified and me unable to implement a fix i'll close this issue. If the problem still persists let's say a week or two from now make another post here and I'll look into a potential workaround.