Incorrect video assembly in some cases

marabu88 commented 4 years ago

Fill in this template completely. Issues not following this template will be closed and ignored.

Check the boxes below by filling `[ ]` with an `x` so it looks like `[x]`.

Use the Preview button to ensure the template is filled in correctly.

[x] I am using the latest version of PyInstaLive: 3.2.3.
[x] I have installed either Python 2.7.x or 3.5+: Python 3.7
[x] I have read the README.
[x] I have read the FAQ.
[x] I have checked all existing issues, none of which could solve my issue.

I am using:

[x] Windows 10
[ ] Windows 8.1
[ ] Windows 8
[ ] Windows 7
[ ] Linux (distribution: )
[ ] macOS (version: )
[ ] Other (device & OS name: )

To report a bug, fill in the information below.

Behavior

There are at least 2 problems that I have noticed recently. One is that the segment records in the json file look like this:

  "segments": {
    "17850663062287134_0-2855467.m4v?ms=m_CN": "540p",
    "17850663062287134_0-2857467.m4v?ms=m_CN": "540p",
    "17850663062287134_0-2859467.m4v?ms=m_CN": "540p",
    ...
  }

The second problem leads to the following result (I hid the path to the file and the name of the account for obvious reasons):

---------------------------------------------------------------------------
[I] PYINSTALIVE (SCRIPT V3.2.3 - PYTHON V3.7.4) - 09-03-2020 04:57:50 AM
---------------------------------------------------------------------------
[I] Assembling video segment files from specified folder: ..._17865730147990241_1599083840_live_downloads
[W] Audio segment not found: ..._17865730147990241_1599083840_live_downloads\17865730147990241_0-68313.m4a
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000027246381240] could not find corresponding track id 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000027246381240] could not find corresponding trex
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000027246381240] error reading header
..._17865730147990241_1599083840_live_downloads\assembled_source_17865730147990241_0_m4a.tmp: Invalid data found when processing input
[W] FFmpeg exit code not '0' but '1'.
---------------------------------------------------------------------------

Steps to reproduce issue

I have no idea why it happens. some threads from the same accounts do not cause such problems. it just happens. if someone can find out the reason, write this one here.

My suggestions

In the first case, I simply added a string to the assembler.py file, the string:

93: segment = re.sub('\?.*$', '', segment)

This simply removes a part of the file name that is not saved on the system, since the "?" character cannot be used in the file name. This fix works fine. I don't know if there is such a problem in operating systems that are not Windows family.

In the second case, I was able to manually build a complete video. so I don't understand why it wasn't done automatically. but I think it's necessary to change the program's work so that in case of an error during video building only temporary files will be deleted, and the segments received from the Internet will remain on the disk, so that it will be possible to build them by themselves. now, regardless of the result of video building, the original files will be deleted if the settings choose to delete the original files.

Jeffade1 commented 4 years ago

I confirm the problem. Since the start of the week, alot of livestreams are with "?ms=m_CN" after ".m4v" of each segment, so it doesn't assemble when it ends and need to be done manually. I think dvingerh can release a new version with a quick fix for this issue.

dvingerh commented 4 years ago

First problem is an issue with filename sanitizing in dependency module, i added regex workaround you provided in latest version bump 3.2.4. Second problem the cause is unknown and i have no way to reliably reproduce the problem so I cant fix it.

In case of segment files being deleted this is also handled in dependency module, if ffmpeg returns error it will not delete files but if it doesnt return error it'll remove files. https://github.com/ping/instagram_private_api_extensions/blob/master/instagram_private_api_extensions/live.py#L490 This project is semi-abandoned but will still accept pull requests with user's own fixes for non-critical issues like these

Jeffade1 commented 4 years ago

Thanks for the responsiveness dvingerh👍 I'm waiting for the .exe version of 3.2.4 to use it.

dvingerh commented 4 years ago

https://github.com/dvingerh/PyInstaLive/releases/tag/3.2.4

marabu88 commented 4 years ago

Today I found that the videos I downloaded were again not collected correctly, although I made corrections. I don't understand how it works, but it only works for me if I manually collect the video using the -as parameter. I'm asking the developer to figure it out. Perhaps I did not fully understand the logic of the program. I think you need to make changes to the place where the json file is written.

p.s. I seem to have understood. In automatic mode, do you use instagram_private_api_extensions to build the video? You can't call the pyinstalive function instead?

Jeffade1 commented 4 years ago

https://github.com/dvingerh/PyInstaLive/releases/tag/3.2.4

After one day of testing, everything seems to be working as good as before😉

dvingerh commented 4 years ago

With the external API dependency essentially abandoned by its author I will have to look into implementing a workaround for the video assembly issues whenever I have enough time and feel like doing so. Could be tomorrow could be next month, I don't know. It's probably not a whole lot of work but I still want to ask others to submit any pull requests should they already have their own workaround.

For clarification @marabu88

Today I found that the videos I downloaded were again not collected correctly, although I made corrections. I don't understand how it works, but it only works for me if I manually collect the video using the -as parameter.

The external dependency normally takes care of video assembly but due to filename sanitation issues it fails to do so correctly. The -assemble command uses PyInstaLive's own implementation of video assembly and is able to both do so with a (not properly sanitized) JSON file and without a JSON file at all. The workaround will consist of not using the external dependency's assembler but instead switch to PyInstaLive's own implementation.

marabu88 commented 4 years ago

I was just using your version

pil.assemble_arg = live_mp4_file.replace(".mp4", "_downloads.json")
assembler.assemble(user_called=False)

instead of

pil.broadcast_downloader.stitch(live_mp4_file, cleartempfiles=pil.clear_temp_files)

but I didn't fix the program code completely, so you have to fix it yourself to avoid duplication.

I don't use developer tools, I just fix the source code of the program, so I try to make minimal changes so as not to break what works :) I hope you find time to do it all nicely.

dvingerh commented 4 years ago

Yeah, that's the gist of it. I'll just have to make sure it all plays nicely with the user settings (e.g. removing temporary files but only when it's successful) and displays the proper console messages

wobbleninja commented 4 years ago

Is this the same issue that the script doesn't merge the files at all? I can't really understand since I don't have the knowledge.. but it started some time ago where only a few of the videos would be merged and now literally I get no video out of it while the script records normally.

I see the folder and the chunks but after the live is over there's no video at all. I only get the json file and no videos at all. Not even a few, just nothing.

Edit: Okay now I'm catching on even with my poor "coding" (not even) skills. I see that the -as command can merge the videos. And since I created my own loop so the script runs again and again in the -df command to record everything I follow, I need to find a way to make a small script in my loop so it runs the -as command on all the "_downloads.json" files and merge all the videos at once (instead of running the -as for each json file) as a fast workaround until the script is fixed.

bamtan commented 4 years ago

@wobbleninja if you're using Linux you could run the following to compile all the videos in the current directory

find * -type f -name "*_live_downloads.json" -exec pyinstalive -as "{}" \;

wobbleninja commented 4 years ago

@bamtan Nah, apparently I am on Windows.

I'm trying something like this

cd <folder with my .json files>
for %%a in (*.json) do ( 
pyinstalive -as %%a
)

And it looks okay so far.

dvingerh / PyInstaLive