search

dvingerh / PyInstaLive

Python script to download Instagram livestreams.
MIT License
547 stars 111 forks source link

Re: Download livestream from the start #121

Closed cykac04 closed 3 years ago

cykac04 commented 3 years ago

ins_v2.txt

Fill in this template completely. Issues not following this template will be closed and ignored.

Check the boxes below by filling [ ] with an x so it looks like [x].

Use the Preview button to ensure the template is filled in correctly.

I am using:

Question

For the previous request of download the previous seconds or minutes of an ongoing instagram live (Download livestream from the start #24), I have written a python script, which I uploaded, to accomplish this by reading the "*downloads.json" file. The instalive stream seems to be started from some "000900" , "000934" or "001001" segment. Then I merge downloaded files manually. Would you be possible to embed such function in your code?

Jeffade1 commented 3 years ago

If this works and that @dvingerh can make a quick integration, it would be fantastic👍 Especially with Instagram rate limits changes since some months that have complicated the permanent scripts.

bamtan commented 3 years ago

Nice. It works, but it is a bit slow.

Final merge command is something like this:

cat init.m4v $(ls -vx  <stream_id>_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx  <stream_id>_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4
cykac04 commented 3 years ago

Nice. It works, but it is a bit slow.

I'm not sure if linear search can use multithread.

I have found that when a user join the livestream, the timestep can be less than 2000 and it will sucks. If this situation is included, the parse will be much slower.

Infinitay commented 3 years ago

Oh wow this is wonderful, I never knew this was possible with IG Lives. This is would amazing to have for livestream downloading via PyInstaLive. Do you know when dash_playback_url expires? Is it immediately after the IG Live or 24 hours later? I know for a fact it expires after over 48-72 hours.

EDIT: Seems like the timeframe is 24 hours. I would also like to note I wasn't able to view the m4vs/m4as like you would be able to with a normal download via PyInstaLive. However, after using bamtans commands followed by merging the concatenated audio/video I was able to play the video with no problem.

cykac04 commented 3 years ago

insaio.txt An aiohttp version of the script. It checks about 150 urls per seconds. Obviously it will take more than hours if you miss thousands of segments.

coder39248583 commented 3 years ago

@cykac04 Once you find the first segment you can find all subsequent segments by the following method:

There may be an even simpler way, but this works. The trick is finding the first segment. I've been researching the dash manifests and there may be a way to get the earliest manifest by including a "start=" query string in the manifest URL, but their time keeping seems wonky. I'll keep looking into it.

cykac04 commented 3 years ago

@cykac04 Once you find the first segment you can find all subsequent segments by the following method:

  • Download init.m4v
  • Find the first segment somehow, i.e. 1200.m4v
  • concat init.m4v and 1200.m4v > init_1200.m4v
  • run: ffprobe -v error -show_entries stream=duration_ts -of default=nw=1:nk=1 init_1200.m4v
  • output: 2401
  • Download 2401.m4v
  • concat init.m4v and 2401.m4v > init_2401.m4v (there is no need to concat earlier segments)
  • run: ffprobe -v error -show_entries stream=duration_ts -of default=nw=1:nk=1 init_2401.m4v
  • output: 2499
  • repeat

There may be an even simpler way, but this works. The trick is finding the first segment. I've been researching the dash manifests and there may be a way to get the earliest manifest by including a "start=" query string in the manifest URL, but their time keeping seems wonky. I'll keep looking into it.

17900932982065128.zip I have a stream getting error. I get 2400 from ffprobe but the correct segment is 2501

M4TCHL3SS commented 3 years ago 
cat init.m4v $(ls -vx  <stream_id>_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx  <stream_id>_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Anyone now how can merge this ? I tried but i think it's not working on windows 10. 18129352513204028.zip

Infinitay commented 3 years ago 
cat init.m4v $(ls -vx  <stream_id>_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx  <stream_id>_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Anyone now how can merge this ? I tried but i think it's not working on windows 10. 18129352513204028.zip

Keep in mind you can't run the commands in Windows CMD or Powershell as is because some of those commands and the syntax is not supported. The easiest thing to do would be to download Git and install it accordingly. Open up Git Bash and make sure your working directory within Git Bash is the same location where those m4v and m4a files are. Then you could run the following snippet, replacing stream ID with your stream id (its the folder name).

cat init.m4v $(ls -vx 18129352513204028_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx 18129352513204028_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4
M4TCHL3SS commented 3 years ago 
cat init.m4v $(ls -vx  <stream_id>_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx  <stream_id>_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Anyone now how can merge this ? I tried but i think it's not working on windows 10. 18129352513204028.zip

Keep in mind you can't run the commands in Windows CMD or Powershell as is because some of those commands and the syntax is not supported. The easiest thing to do would be to download Git and install it accordingly. Open up Git Bash and make sure your working directory within Git Bash is the same location where those m4v and m4a files are. Then you could run the following snippet, replacing stream ID with your stream id (its the folder name).

cat init.m4v $(ls -vx 18129352513204028_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx 18129352513204028_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Thank you. I already have git and It's working perfectly right now.

coder39248583 commented 3 years ago

@cykac04 That is odd. How much time elapsed after the live ended before you attempted to download the segments? Did the ffprobe method work for the other segments? Did the 2400.mkv link return 404? Did you try the 2400 link without leading 0s? i.e. 2400 instead of 0002400 If downloaded, was it a 0 byte file? Did you try slightly less or more than 2400? Such as 2399? 2398? I'll try to keep an eye out for this problem and see if I can learn more. Until then you can use the ffprobe method as a first resort and fall back on brute force if you encounter problem segments.

cykac04 commented 3 years ago

How much time elapsed after the live ended before you attempted to download the segments?

It started with pyinstalive.

Did the ffprobe method work for the other segments?

Yes.

Did the 2400.mkv link return 404?

Yes.

Did you try the 2400 link without leading 0s? i.e. 2400 instead of 0002400

No, but it should be the same.

If downloaded, was it a 0 byte file?

Yes.

Did you try slightly less or more than 2400? Such as 2399? 2398?

I brute force it, nothing found.

coder39248583 commented 3 years ago

@cykac04 After 2400 failed did your brute force method begin at 2401 or go back and start at 935? When you merged the files, was there a jump in video at the 934-2501 boundary? The video segments in your zip are black for me, and I don't have the audio files, and if they're silent for you then I guess we can't tell, but if the error ever happens while there's video or audio playing then it would be good to know. Thanks for answering all my questions.

cykac04 commented 3 years ago

@cykac04 After 2400 failed did your brute force method begin at 2401 or go back and start at 935?

all numbers until the start of recording

When you merged the files, was there a jump in video at the 934-2501 boundary?

I don't realize any gap.

Thanks for answering all my questions.

coder39248583 commented 3 years ago

@cykac04

all numbers until the start of the recording

Forgive me for being confused on this point, but you brute forced it backwards? Or, when 2400 failed, did you brute force it from 935 onwards?

I don't realize any gap.

I guess there's audio playing? I don't have the audio segments, so I can't tell. But if it's just black and silent then there's no way to know. If the error happens while there's video or audio activity then it'll be something to look for.

challi1337 commented 3 years ago 
cat init.m4v $(ls -vx  <stream_id>_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx  <stream_id>_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Anyone now how can merge this ? I tried but i think it's not working on windows 10. 18129352513204028.zip

Keep in mind you can't run the commands in Windows CMD or Powershell as is because some of those commands and the syntax is not supported. The easiest thing to do would be to download Git and install it accordingly. Open up Git Bash and make sure your working directory within Git Bash is the same location where those m4v and m4a files are. Then you could run the following snippet, replacing stream ID with your stream id (its the folder name).

cat init.m4v $(ls -vx 18129352513204028_0-*.m4v) > video.mp4 && 
cat init.m4a $(ls -vx 18129352513204028_0-*.m4a) > audio.m4a && 
ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4

Thanks for the help I used to log on a linux machine just to merge them.

I have one issue, the command as it stands "cat init.m4v $(ls -vx 17880274787491353_0-.m4v) > video.mp4 && cat init.m4a $(ls -vx 17880274787491353_0-.m4a) > audio.m4a && ffmpeg -i video.mp4 -i audio.m4a -c copy output.mp4"

will end up with this: image

I can only merge if I use the 3 lines separately. Do you know if there is a way to make it work in one go?

Also is there a possibility for omitting the stream ID and just let it merge .m4v and .m4a files in the order its listen on the file system?

Edit: I'm using the pyinstalive docker on my Synology NAS and for some reason it doesn't merge the downloaded files without any error message or anything, if anyone knows what the issue could be, much appreciated.

dvingerh commented 3 years ago

I am willing to implement this if you can figure out a more efficient way to find and download the missing segments instead of brute-forcing thousands of URLs.

In other news I have a testing branch which authenticates using and interacts with the Instagram web API and no longer uses the very much outdated app API dependency. Its extension module is still used to handle downloading of segments but I don't expect the underlying logic to change in the foreseeable future

You can test with pip install git+https://github.com/dvingerh/PyInstaLive.git@testing --upgrade

ghost commented 3 years ago

Am I using this script correctly? Just place it in a folder with the relevant .json and let it run?

dvingerh commented 3 years ago

This issue has ran its course for now I will be closing it. @Testing2134 For any questions please create a new issue.

babyrig commented 3 years ago

You can test with pip install git+https://github.com/dvingerh/PyInstaLive.git@testing --upgrade

Hi sorry to disturb you just a little question about this testing version: You are running PyInstaLive 3.3.0 using Python 3.8.10

The final file/live have a low quality compared to the old/stable version, it's suppose to be this way or I am doing something wrong here ?

dvingerh commented 3 years ago

You can test with pip install git+https://github.com/dvingerh/PyInstaLive.git@testing --upgrade

Hi sorry to disturb you just a little question about this testing version: You are running PyInstaLive 3.3.0 using Python 3.8.10

The final file/live have a low quality compared to the old/stable version, it's suppose to be this way or I am doing something wrong here ?

There should be no difference since the way livestreams are downloaded are not changed. Unless all new downloaded streams are lower quality suddenly i would chalk it up to coincidence In any case more testing is needed and these new changes are unlikely to get pushed out as an official new version any time soon, if ever

babyrig commented 3 years ago

There should be no difference since the way livestreams are downloaded are not changed. Unless all new downloaded streams are lower quality suddenly i would chalk it up to coincidence

It's weird the old version running is like 10 times more big the file this testing it's like low quality than the normal I am testing with both versions the same livestream. Eg this is from the testing version the length is little less due I forgot to start at the same time but it's just few minutes less:

image

This one is from the old version (You are running PyInstaLive 3.2.4 using Python 3.6.9):

image

There is any place to change the quality in this testing version ? Right now are recording a livestream and I can see from both the temporary files size differences old version using du -sh: 531M ../IG/ and the testing version: 96M ../IG/

Maybe it's just me and I need to test the same version on this machine then to see if will be fine.

dvingerh commented 3 years ago

@babyrig It was not just you, while rewriting some API endpoints I had it grab the wrong stream dash url I have made a new commit you can test

babyrig commented 3 years ago

I have made a new commit you can test

Now 100% just fine thank you

cykac04 commented 3 years ago

inpst.txt ins+aio.txt

@dvingerh @coder39248583

Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json.

run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time.

run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it.

Please change those "path" and "directory" to what you use.

ghost commented 3 years ago

[W] No matching JSON file found for the segment directory, trying to continue without it. [E] Could not create video file: 'broadcast_dict'

when trying to merge? Is this because it was before I upgraded to testing branch?

dvingerh commented 3 years ago

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

ghost commented 3 years ago

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

I only switched for testing the "download streams from start" bit

cykac04 commented 3 years ago

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

I found it is easily getting challenged after downloaded by using -df in old branch. Is it related to the outdated app API dependency?

ghost commented 3 years ago

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

I found it is easily getting challenged after downloaded by using -df in old branch. Is it related to the outdated app API dependency?

Are saying you get less insta challenges on testing branch? If that's case I'm not going back to master lol

dvingerh commented 3 years ago

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

I only switched for testing the "download streams from start" bit

The testing branch does not have that built-in, the only major change it has is that it no longer relies on an outdated module to authenticate and retrieve livestream information. For the code related to downloading "from start" you can find the link to ins_v2.txt all the way at the top of this issue

Old files including JSON files are not compatible with the testing branch As it stands the master branch still works just fine so I would only switch to the testing branch if major functions stop working

I found it is easily getting challenged after downloaded by using -df in old branch. Is it related to the outdated app API dependency?

Could very well be, your guess is as good as mine. I don't really use this script anymore

mb9023 commented 3 years ago

inpst.txt ins+aio.txt

@dvingerh @coder39248583

Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json.

run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time.

run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it.

Please change those "path" and "directory" to what you use.

@cykac04 I've mostly gotten this to work but I'm getting an error with the inpst.py

[I] Launched finish command: python inpst.py Traceback (most recent call last): File "inpst.py", line 44, in <module> folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1] IndexError: list index out of range

Seems like it's having trouble finding the {user} name, as the path it's looking at is correct and is the same in the other script. In my instalives I have a 20211104_<username>_17920175195000447_1636007477_live_downloads folder and also a matching 17920175195000447 folder. right now it creates a full Live file for the stream I captured, and I have the "beginning of stream" files, but it's not combining them.

If I manually change {user} in the errored line of the script to the actual IG username and run it, it does work and creates a full *_live_pst.mp4 file which seems fine. I'm not sure why it's not working automatically though. atm I have to change it manually for each different user I get a stream for so it will combine them.

cykac04 commented 3 years ago

inpst.txt ins+aio.txt @dvingerh @coder39248583 Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json. run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time. run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it. Please change those "path" and "directory" to what you use.

@cykac04 I've mostly gotten this to work but I'm getting an error with the inpst.py

[I] Launched finish command: python inpst.py Traceback (most recent call last): File "inpst.py", line 44, in <module> folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1] IndexError: list index out of range

Seems like it's having trouble finding the {user} name, as the path it's looking at is correct and is the same in the other script. In my instalives I have a 20211104_<username>_17920175195000447_1636007477_live_downloads folder and also a matching 17920175195000447 folder. right now it creates a full Live file for the stream I captured, and I have the "beginning of stream" files, but it's not combining them.

If I manually change {user} in the errored line of the script to the actual IG username and run it, it does work and creates a full *_live_pst.mp4 file which seems fine. I'm not sure why it's not working automatically though. atm I have to change it manually for each different user I get a stream for so it will combine them.

It is probably the the ps command does not work. 'ps -p {os.getppid()} -o cmd=' This works in linux, and I have got a reply that 'cmd' key does not work in OSX. Of course, windows does not have ps at all. You may have to find the alternative command or just paste the folder as argument.

mb9023 commented 3 years ago

inpst.txt ins+aio.txt @dvingerh @coder39248583 Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json. run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time. run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it. Please change those "path" and "directory" to what you use.

@cykac04 I've mostly gotten this to work but I'm getting an error with the inpst.py [I] Launched finish command: python inpst.py Traceback (most recent call last): File "inpst.py", line 44, in <module> folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1] IndexError: list index out of range Seems like it's having trouble finding the {user} name, as the path it's looking at is correct and is the same in the other script. In my instalives I have a 20211104_<username>_17920175195000447_1636007477_live_downloads folder and also a matching 17920175195000447 folder. right now it creates a full Live file for the stream I captured, and I have the "beginning of stream" files, but it's not combining them. If I manually change {user} in the errored line of the script to the actual IG username and run it, it does work and creates a full *_live_pst.mp4 file which seems fine. I'm not sure why it's not working automatically though. atm I have to change it manually for each different user I get a stream for so it will combine them.

It is probably the the ps command does not work. 'ps -p {os.getppid()} -o cmd=' This works in linux, and I have got a reply that 'cmd' key does not work in OSX. Of course, windows does not have ps at all. You may have to find the alternative command or just paste the folder as argument.

I'm using Powershell to SSH into an Ubuntu VM and running scripts there so seems odd.

cykac04 commented 3 years ago

inpst.txt ins+aio.txt @dvingerh @coder39248583 Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json. run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time. run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it. Please change those "path" and "directory" to what you use.

@cykac04 I've mostly gotten this to work but I'm getting an error with the inpst.py [I] Launched finish command: python inpst.py Traceback (most recent call last): File "inpst.py", line 44, in <module> folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1] IndexError: list index out of range Seems like it's having trouble finding the {user} name, as the path it's looking at is correct and is the same in the other script. In my instalives I have a 20211104_<username>_17920175195000447_1636007477_live_downloads folder and also a matching 17920175195000447 folder. right now it creates a full Live file for the stream I captured, and I have the "beginning of stream" files, but it's not combining them. If I manually change {user} in the errored line of the script to the actual IG username and run it, it does work and creates a full *_live_pst.mp4 file which seems fine. I'm not sure why it's not working automatically though. atm I have to change it manually for each different user I get a stream for so it will combine them.

It is probably the the ps command does not work. 'ps -p {os.getppid()} -o cmd=' This works in linux, and I have got a reply that 'cmd' key does not work in OSX. Of course, windows does not have ps at all. You may have to find the alternative command or just paste the folder as argument.

I'm using Powershell to SSH into an Ubuntu VM and running scripts there so seems odd.

Is the run_at_start script work? Only the finish script is in trouble?

mb9023 commented 3 years ago

inpst.txt ins+aio.txt @dvingerh @coder39248583 Hi all, I've make some updates. Now the scripts will get the PPID, extract the user, and open the corresponding json. run_at_start = python3 ins+aio.py The "start" script only use brute force when the ffprobe method does not work. I've test it can parse up to "7000XXX" segments in much shorter time. run_at_finish = python3 inpst.py The "finish" script will merge the missing segment to the downloaded stream automatically. Else, it would redownload the pst quality video when pyinstalive sometimes does not download it. Please change those "path" and "directory" to what you use.

@cykac04 I've mostly gotten this to work but I'm getting an error with the inpst.py [I] Launched finish command: python inpst.py Traceback (most recent call last): File "inpst.py", line 44, in <module> folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1] IndexError: list index out of range Seems like it's having trouble finding the {user} name, as the path it's looking at is correct and is the same in the other script. In my instalives I have a 20211104_<username>_17920175195000447_1636007477_live_downloads folder and also a matching 17920175195000447 folder. right now it creates a full Live file for the stream I captured, and I have the "beginning of stream" files, but it's not combining them. If I manually change {user} in the errored line of the script to the actual IG username and run it, it does work and creates a full *_live_pst.mp4 file which seems fine. I'm not sure why it's not working automatically though. atm I have to change it manually for each different user I get a stream for so it will combine them.

It is probably the the ps command does not work. 'ps -p {os.getppid()} -o cmd=' This works in linux, and I have got a reply that 'cmd' key does not work in OSX. Of course, windows does not have ps at all. You may have to find the alternative command or just paste the folder as argument.

I'm using Powershell to SSH into an Ubuntu VM and running scripts there so seems odd.

Is the run_at_start script work? Only the finish script is in trouble?

Yes the start script works, I've got the {id} folder with the missed stream files. Only the finish script errors.

EDIT: Your comment made me check the other script and I realized that for some reason my user = re.search(r'(?<=-d\s)[^\s]+', output).group(0) line was missing the .group(0) in the finish script. Now if I try to run it manually I get

Traceback (most recent call last): File "inpst.py", line 43, in <module> user = re.search(r'(?<=-d\s)[^\s]+', output).group(0) AttributeError: 'NoneType' object has no attribute 'group'

Seems like this is just because re.search sees nothing in the log as there wasn't a stream

cykac04 commented 3 years ago

Yes the start script works, I've got the {id} folder with the missed stream files. Only the finish script errors.

It sounds weird as the both scripts are the same. May be you can try this. run_at_finish = bash -c 'python3 -u /home/madoka/Command/inpst.py $(ps -p $PPID -o cmd=)'

if arg == 1:
    folder = f'{path}/instalives/{sys.argv[1]}'
elif arg == 9:
    #output = check_output(f'ps -p {os.getppid()} -o cmd=', shell=True, encoding='UTF-8')
    user = sys.argv[4]#re.search(r'(?<=-d\s)[^\s]+', output).group(0)
    folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1]
    print(f'Getting info for \'{user}\' successful.\n')
else:
    exit()

But remind that arg ==9 is only for using -df option. python3 pyinstalive -d <user> -cp pyinstalive.ini -dp instalives/ --no-replays You have to change it according to what command you use.

mb9023 commented 3 years ago

Yes the start script works, I've got the {id} folder with the missed stream files. Only the finish script errors.

It sounds weird as the both scripts are the same. May be you can try this. run_at_finish = bash -c 'python3 -u /home/madoka/Command/inpst.py $(ps -p $PPID -o cmd=)'

if arg == 1:
    folder = f'{path}/instalives/{sys.argv[1]}'
elif arg == 9:
    #output = check_output(f'ps -p {os.getppid()} -o cmd=', shell=True, encoding='UTF-8')
    user = sys.argv[4]#re.search(r'(?<=-d\s)[^\s]+', output).group(0)
    folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1]
    print(f'Getting info for \'{user}\' successful.\n')
else:
    exit()

But remind that arg ==9 is only for using -df option. python3 pyinstalive -d <user> -cp pyinstalive.ini -dp instalives/ --no-replays You have to change it according to what command you use.

Your comment made me check the other script and I realized that my user = re.search(r'(?<=-d\s)[^\s]+', output) line was missing the .group(0) in the finish script. But now if I try to run it manually I get-

Traceback (most recent call last): File "inpst.py", line 43, in <module> user = re.search(r'(?<=-d\s)[^\s]+', output).group(0) AttributeError: 'NoneType' object has no attribute 'group'

I also have to remove the .group(0) part of the line for the script to work when I replace {user} with the directly, or it just gives the same error message.

I just had another livestream today I was recording and the run_at_finish script errored with this again. I'm not sure where exactly this line is looking to find the username. My command is simply pyinstalive -d username.

Your script edit is giving me this error [I] Launched finish command: bash -c 'python3 -u /mnt/share/pyinstalive/inpst.py $(ps -p $PPID -o cmd=)' Traceback (most recent call last): File "/mnt/share/pyinstalive/inpst.py", line 39, in <module> if arg == 1: NameError: name 'arg' is not defined

cykac04 commented 3 years ago

@mb9023 It sounds like ps does not work well get the command. I am using this version. ps -V ps from procps-ng UNKNOWN You have check whether your ps allows -p option and cmd key.

mb9023 commented 3 years ago

@mb9023 It sounds like ps does not work well get the command. I am using this version. ps -V ps from procps-ng UNKNOWN You have check whether your ps allows -p option and cmd key.

@cykac04 Mine is the same as yours.

I re-downloaded the inspst.txt file from this thread and replaced my script and it's partly working now, so I must have messed something up before.

However, I am not getting the final _pst.mp4 file. I am getting a *_live.mp4 and a *_live-.mp4. The *user*live_downloads folder is getting deleted, I also do not see an {id}_pst folder that should be created. It seems like the second half of the script is not doing anything. There are no error messages.

My ins_pst.log looks like this

Getting info for {user} successful. PST stream was already downloaded. Merging downloaded files.

cykac04 commented 3 years ago

@mb9023 It sounds like ps does not work well get the command. I am using this version. ps -V ps from procps-ng UNKNOWN You have check whether your ps allows -p option and cmd key.

@cykac04 Mine is the same as yours.

I re-downloaded the inspst.txt file from this thread and replaced my script and it's partly working now, so I must have messed something up before.

However, I am not getting the final _pst.mp4 file. I am getting a *_live.mp4 and a *_live-.mp4. The *user*live_downloads folder is getting deleted, I also do not see an {id}_pst folder that should be created. It seems like the second half of the script is not doing anything. There are no error messages.

My ins_pst.log looks like this

Getting info for {user} successful. PST stream was already downloaded. Merging downloaded files.

Well, you should read the words. The api got pst quality so it will just merge the segments.

mb9023 commented 3 years ago

Well, you should read the words. The api got pst quality so it will just merge the segments.

@cykac04 I don't know what pst quality is, I assumed that was supposed to just be the final output. Is it not supposed to merge together the missed stream files with the live captured stream files to create one final? I am getting two files and I have to manually combine them to get a full live still

cykac04 commented 3 years ago

Well, you should read the words. The api got pst quality so it will just merge the segments.

@cykac04 I don't know what pst quality is, I assumed that was supposed to just be the final output. Is it not supposed to merge together the missed stream files with the live captured stream files to create one final? I am getting two files and I have to manually combine them to get a full live still

PST is the original quality with highest bitrate, but may not be the highest resolution. _live.mp4 is what pyinstalive downloaded. _live-.mp4 is missing segments + *_live.mp4.

mb9023 commented 3 years ago

PST is the original quality with highest bitrate, but may not be the highest resolution. _live.mp4 is what pyinstalive downloaded. _live-.mp4 is missing segments + *_live.mp4.

My *_live-.mp4 is missing segments ONLY. It's 16 seconds long while the full test stream was almost 3 minutes.

cykac04 commented 3 years ago

PST is the original quality with highest bitrate, but may not be the highest resolution. _live.mp4 is what pyinstalive downloaded. _live-.mp4 is missing segments + *_live.mp4.

My *_live-.mp4 is missing segments ONLY. It's 16 seconds long while the full test stream was almost 3 minutes.

Oops. Just add something to the old code. import time and time.sleep(300) before print('PST stream was already downloaded.\nMerging downloaded files.') It becase it merged before "pyinstalive" finished everything.

yoshyv commented 3 years ago

I got both errors as @mb9023 

Traceback (most recent call last):
  File "ins+aio.py", line 92, in <module>
    j_file = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads.json'), key=os.path.getctime)[-1]
IndexError: list index out of range
Traceback (most recent call last):
  File "inpst.py", line 44, in <module>
    folder = sorted(glob.glob(rf'{path}/instalives/*{user}*downloads'), key=os.path.getctime)[-1]
IndexError: list index out of range

so i looked up the scripts and realized my pyinstalive didn't create the instalives subdirectory while downloading live streams. There's no such option in pyinstalive.sh why do you have this subdirectory?

Then I edited all paths in the scripts and manually ran ins+aio.py while the stream was still live and got something similar to

Traceback (most recent call last): File "inpst.py", line 43, in user = re.search(r'(?<=-d\s)[^\s]+', output).group(0) AttributeError: 'NoneType' object has no attribute 'group'

(not exactly the same cuz it's an ins+aio.py error )

How do I solve this problem and can I still download the whole stream? The livestream is over now. I'm running the pyinstalive and the script on Ubuntu 20.04 and

You are running PyInstaLive 3.2.4 using Python 3.8.10

mb9023 commented 3 years ago

Testing on short streams seems to be working now! I'm really not sure what happened to the script I had that gave me so many issues.. I basically ended up putting everything back to how it was originally as well as adding the time.sleep seemed to help (though I lowered it to 10 seconds instead of 300).

@yoshyv I just created the instalives folder myself and pointed my pyinstalive.ini script to save streams there. The other errors I think you're basically just always going to get those running manually as it's looking for the active stream ID. I had to use a second IG account to go live on to keep testing my captures. I'm not sure about capturing streams that have ended.

cykac04 commented 3 years ago

@yoshyv run the script manually after you confirm the path is correct. python3 ins+aio.py 20211108_felonyrose__n_17953574011501851_1636370890_live_downloads Definitely all your errors are not finding user from ps command.

cykac04 commented 3 years ago

Testing on short streams seems to be working now! I'm really not sure what happened to the script I had that gave me so many issues.. I basically ended up putting everything back to how it was originally as well as adding the time.sleep seemed to help (though I lowered it to 10 seconds instead of 300).

@yoshyv I just created the instalives folder myself and pointed my pyinstalive.ini script to save streams there. The other errors I think you're basically just always going to get those running manually as it's looking for the active stream ID. I had to use a second IG account to go live on to keep testing my captures. I'm not sure about capturing streams that have ended.

300 is actually for long instalive which may merge for more minutes.

yoshyv commented 3 years ago

@yoshyv run the script manually after you confirm the path is correct. python3 ins+aio.py 20211108_felonyrose__n_17953574011501851_1636370890_live_downloads Definitely all your errors are not finding user from ps command.

Problem is, the stream is ended now and fragments (and the directory) are merged then deleted. Am I still able to retrieve the lost part?