Handle bluray folders - Githubissues

breezytm commented 5 years ago

Hello Clinton,

Is there a way for nzbToMedia to transcode the bluray to mkv?

clinton-hall commented 5 years ago

I thought it did...

is this just the folder structure, or an iso?

clinton-hall commented 5 years ago

I have added the basics for this... but don't have anything to test it on here at the moment. If you want to test, switch to bluray-1 branch and test it out.

breezytm commented 5 years ago

I will go head and switch my branch to nightly and test the changes you made. It is interesting that I had one iso that worked fine and two others failed. perhaps it could be folder structure difference. Please let me know if I should send you the folder structure of one of the failed one.

clinton-hall commented 5 years ago

Yes, any detaiks as to folder structure would help.

Please use bluray-1 branch, not nightly...

breezytm commented 5 years ago

I have not been able to switch to the latest branch but here's a log of one of the issues. It's not always the same. https://ufile.io/kq4ez

:/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$ ls A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel.iso :/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$

The file is still an iso. It wasn't transcoded into an MKV

clinton-hall commented 5 years ago

Yes, the issue here is that there was a bug if you have "Concat" enabled and no .vob files are found.

This is fixed in bluray-1 branch (along with checking for bluray structure in addaition to DVD .vob structures.

You will need to switch to bluray-1 branch to test this. I can guarantee that nightly branch will not currently trasncode any bluray images.

breezytm commented 5 years ago

I am about to test this right now. I'll report back with my findings.

breezytm commented 5 years ago

What am I doing wrong?

admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git checkout bluray-1 Branch bluray-1 set up to track remote branch bluray-1 from origin. admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git pull remote: Enumerating objects: 8, done. remote: Counting objects: 100% (8/8), done. remote: Compressing objects: 100% (8/8), done. remote: Total 8 (delta 2), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (8/8), done. From https://github.com/clinton-hall/nzbToMedia 82d1ba8..0a3b2cd bluray-1 -> origin/bluray-1 cb3f61f..825b48a nightly -> origin/nightly Updating 82d1ba8..0a3b2cd Fast-forward core/transcoder.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

https://ufile.io/5xze2

breezytm commented 5 years ago

Also can you merge the HVEC change to this branch as well. I could test them both in this use case.

clinton-hall commented 5 years ago

ok, I have merged h265 changes into bluray-1 You have correctly switched branch probably worth doing another sudo git pull and then all should be ready to test

breezytm commented 5 years ago

What I was saying is I switched branch but it still failed. log file is attached at the bottom of my previous comment.

clinton-hall commented 5 years ago

ok, I see the error now. At least it isn't crashing, and now just says it cent find matching files to process.

What I need to see is just what the file structure is in that .iso....

I'll see tonight if I can download the same iso and check this to find out what it isn't matching.... are you able to email the .nzb file to fock_wulf@hotmail.com?

breezytm commented 5 years ago

Sure. Emailing right now

clinton-hall commented 5 years ago

thanks.... in about 7 hours I should be able to start testing!

Hopefully I'll get some time over the weekend to debug the bluray parsing and get this working for you.

clinton-hall commented 5 years ago

Does that image actually work to be played by bluray players (e.g. in kodi)?

When I use 7zip to view the contents, there is a com directory, a META-INF, and a kudu directory along with a whole stack of *.class files

There is not a single .mts file, there is no BDMV or SOURCE directory... so as it stands nzbToMedia can't do anything with this.

All searching I have done indicates that Bluray folder structure should have a BDMV and a CERTIFICATE directory at the root level.

https://en.wikipedia.org/wiki/Blu-ray#Directory_and_file_structure

So there could be a couple of things going on here:

1. This .iso is not a proper bluray image.
1. The bluray .iso is somehow encrypted and needs to be decrypted before the actual folder structure can be properly parsed.
1. 7zip can't read bluray .iso files correctly.

clinton-hall commented 5 years ago

ok... this is a limitation with 7zip.... (And I updated to v16.02) When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

clinton-hall commented 5 years ago

Ok... I just tried using https://github.com/barneygale/isoparser

isoparser/source.py", line 93, in unpack_volume_descriptor
    raise SourceError("Wrong volume descriptor identifier")
isoparser.source.SourceError: Wrong volume descriptor identifier

I am wondering if we need to fall-back to mounting the iso and then parsing the contents.

breezytm commented 5 years ago

ok... this is a limitation with 7zip.... (And I updated to v16.02) When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

That was me :) https://github.com/clinton-hall/nzbToMedia/issues/1302

clinton-hall commented 5 years ago

LOL... well, aren't you giving me a headache :P ? Thanks for linking that issue back in. I knew I had looked at these...

So I can now confirm that isoparser does NOT support bluray iso files. I have not had a chance to test https://github.com/clalancette/pycdlib yet

breezytm commented 5 years ago

LOL this is me helping you helping me helping everyone if that makes sense.

breezytm commented 5 years ago

In all fairness I don't believe that particular iso is a valid one. MakeMKV can read it though.

clinton-hall commented 5 years ago

What I will like do is keep the 7zip parsing (if it works, it is efficient) but then if that doesn't work, try a mount and parse (as a fall-back).

The only other things I'll need to check is if I need to try and make this smarter and actually read the various play-list files.... current methodology is to transcode all .mts files into a single video file. but that may very well see all additional materials embedded before.after the main feature movie...

clinton-hall commented 5 years ago

Ok. I have added the fall-back to mounting .iso files to parse (only for linux at this time). This is very theoretical, as I am not able to specifically test this functionality (working remotely).

I would be fascinated to see the output of logging from the script against this same download (after updating to latest bluray-1 branch).

If it transcodes, the big question will be what does the video file look like? i.e. is the movie continuous with extra features added at the end, or is it all jumbled and out of order?

If it does not transcode, what does the logging show us to reveal what I have missed?

clinton-hall commented 5 years ago

as of this moment, using bluray-1 I am transcoding that very .iso :) So that part is working.... once it has transcoded I'll then need to verify what the movie itself looks like

clinton-hall commented 5 years ago

wow... this has been transcoding for over 11 hours! The output video is only up to 3 GB!

So it appears to be working... but testing on a NAS is slow going. I still need to verify just how the transcoded video looks and if it it continuous.

breezytm commented 5 years ago

Something is not right. MakeMKV does all of that in under 30 minutes for me but then again I am using a computer with decent specs. Let me use my server to test. I should post process much quicker. I am also wondering if it is transcoding the video as well since it is HVEC. I know you were supposed to merge the changes. Maybe it isn't working correctly. That could explain the longer transcoding time.

breezytm commented 5 years ago

Before I even redownload the file, can you please confirm that when I select MKV as the outputDefaultk, it would copy the video instead of transcoding it as requested in this thread https://github.com/clinton-hall/nzbToMedia/issues/1592

clinton-hall commented 5 years ago

/usr/local/bin/ffmpeg -loglevel warning -fflags +genpts -i concat:/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00002.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00007.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00008.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00022.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00276.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00277.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00278.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00279.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00280.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00281.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/00282.m2ts||/volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp/BDMV/STREAM/0028

So I believe the issue here is that it hasn't specified the output name, or what codec etc... looks like a limitation in the length of the command???

breezytm commented 5 years ago

I am testing using latest pull 5 minutes ago with Dunkirk 2017 MULTI REMUX 2160p 10bit BluRay UHD HDR HEVC DTS-HD MA 7 1-LEGi0N for testing purposes. The name isn't as long so let's see.

clinton-hall commented 5 years ago

at my end definitely issues with the length of the command. The log doesn't capture the full length, and nor does the actual ffmpeg process...

clinton-hall commented 5 years ago

ok... I have figured out how to pipe the videos into ffmpeg, so I can not worry about the length of the command...

but it looks like it is trying to transcode the video

/usr/local/bin/ffmpeg -loglevel warning -fflags +genpts -i - -map 0:0 -c:v libx264 -sn /volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.mkv

it also appears to not be transcoding audio...

breezytm commented 5 years ago

ok... I have figured out how to pipe the videos into ffmpeg, so I can not worry about the length of the command...

but it looks like it is trying to transcode the video
/usr/local/bin/ffmpeg -loglevel warning -fflags +genpts -i - -map 0:0 -c:v libx264 -sn /volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.mkv
it also appears to not be transcoding audio...

Shouldn't you need libx265 instead of libx264 to keep it from transcoding the video and just copy it instead? http://trac.ffmpeg.org/wiki/Encode/H.265

clinton-hall commented 5 years ago

I have got something working now...

/usr/local/bin/ffmpeg -loglevel warning -fflags +genpts -i - -map 0:0 -map 0:2 -c:v copy -c:a:0 copy -sn /volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.mkv

major changes, once I have tested I'll commit these changes I have added a test to each .m2ts file to see if it has audio and video (menu clips don't necessarily have both) and then combine everything that has audio and video.

clinton-hall commented 5 years ago

[14:56:37] [INFO]::MAIN: Transcoding of video to /volume1/Download/NZBGet/dst/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.mkv succeeded
--

15 minutes... much better. Just going to check the video now

breezytm commented 5 years ago

My server is remote and I lost connection to it. So now it's going to take some time for me to test but anyway, would you create a profile call MKV-Bluray or something because since you are doing a copy, what happened to people who may want to convert x265 to x264 since their hardware can not handle it.

clinton-hall commented 5 years ago

I have just pushed my changes... will be interesting to see what this does at your end.

For me this transcoded in 15 minutes. The movie plays and looks good, but I don't have audio at my end (ffprobe says there is audio, but my kodi is not detecting it)

clinton-hall commented 5 years ago

added a new default mkv-bluray https://github.com/clinton-hall/nzbToMedia/commit/13523ea8758c08c3006def430c962d0bc6d121ab

clinton-hall commented 5 years ago

ok... I take it back... my transcoded file has sound... my NAS can't keep up to play the video file since it is too high resolution and so it buffers and I get little sound...

I do note the movie is shown as 3 hours long... so i assume this does indeed have all extra bonus features appended to the end.

breezytm commented 5 years ago

Try mounting the share to a computer and try playing it using VLC. It should give you access to the tracks and hopefully the chapters as well.

Server is back up. Starting right now. I'll let you know the end result.

clinton-hall commented 5 years ago

tried with VLC... too much buffering (only connected wirelessly). I'd need to transfer the whole file to my PC which would take too long... otherwise I would need to use ffmpeg to down-scale to x264 so that I could actually play it with my hardware.

breezytm commented 5 years ago

No need. I can test. It is downloading again. I had to delete it because I ran out of disk space hence why my server crashed. These downloads are huge but anyway, I'll let you know my findings.

Did it keep the chapters? You should be able to see that with VLC

breezytm commented 5 years ago

Failed for me

https://ufile.io/5rcii

breezytm commented 5 years ago

And yes I did do a pull and I did select mkv-bluray

remote: Enumerating objects: 26, done. remote: Counting objects: 100% (26/26), done. remote: Compressing objects: 100% (18/18), done. remote: Total 26 (delta 11), reused 14 (delta 8), pack-reused 0 Unpacking objects: 100% (26/26), done. From https://github.com/clinton-hall/nzbToMedia 066e93e..13523ea bluray-1 -> origin/bluray-1 bbcef52..0ba4b9d nightly -> origin/nightly Updating 066e93e..13523ea Fast-forward autoProcessMedia.cfg.spec | 2 +- core/init.py | 10 +++++++++- core/transcoder.py | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------ nzbToCouchPotato.py | 2 +- nzbToLidarr.py | 2 +- nzbToMedia.py | 2 +- nzbToNzbDrone.py | 2 +- nzbToRadarr.py | 2 +- nzbToSickBeard.py | 2 +- 9 files changed, 84 insertions(+), 27 deletions(-)

clinton-hall commented 5 years ago

ok... I am guessing this didn't mount the .iso

Can you see any files in /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp?

what happens if you run the following command?

 mount -o loop /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp

breezytm commented 5 years ago

that folder temp seems to be read only

WARNING: device write-protected, mounted read-only.

breezytm commented 5 years ago

:sudo mount -o loop /opt/nzbget/downloads/completed/movie/Val erian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and .the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downl oads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-w ezjhOurBits/temp [sudo] password for system-admin: mount: /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray .HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp: WARNING: device write-protected, mounted read-only. :/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$

clinton-hall commented 5 years ago

so this mounts ok when you use sudo, but not otherwise?

breezytm commented 5 years ago

See below

`:/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.US A.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$ mount -o loop /opt/nzbget/downloads/completed/movie/Valerian .and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the. City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/ completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhO urBits/temp

mount: only root can use "--options" option `

clinton-hall commented 5 years ago

ok... I see 2 ways of working through this: -1. create a python mount using ctypes.... this may or may not work without using root e.g.

import ctypes
import ctypes.util
import os

libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
libc.mount.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_ulong, ctypes.c_char_p)

def mount(source, target, fs, options=''):
  ret = libc.mount(source, target, fs, 0, options)
  if ret < 0:
    errno = ctypes.get_errno()
    raise OSError(errno, "Error mounting {} ({}) on {} with options '{}': {}".
     format(source, fs, target, options, os.strerror(errno)))

mount('/dev/sdb1', '/mnt', 'ext4', 'rw')

-2. create an option for users to enter root password into config, and call sudo

clinton-hall commented 5 years ago

if you update to latest then edit autoProcessMedia.cfg

[General]
    sudo_password =

enter the sudo password and then test to see if this works to mount and transcode.

clinton-hall / nzbToMedia

Handle bluray folders #1588