USDX on Raspberry Pi

[manureini]

Hello =)

I would really like to run USDX on my Raspberry Pi. With the guide http://forum.ultra-star.de/viewtopic.php?f=13&t=11234 and some other tutorials USDX is now running.

My problem ist that I don't get hardware acceleration in ultrastar. If I use ffplay -vcodec mpeg4_mmal test.mp4 the video is running fine. But the same video is not running in ultrastar with hardware acceleration.

With the command ffplay test.mp4 hardware acceleration is not working, too.

So it is possible to specify the vcodec or something like the ffmpeg decoder(?) in USDX?

If not could someone give me the code to set it? I could compile and test it. So maybe ultrastar is working on a pi one day :P

[basisbit]

When not specifying a decoder, ffmpeg chooses "best" one itself. If that choice turns out to be unusable for you, please create a bugreport at the ffmpeg project, so they can actually fix their bug.

[basisbit]

please add a link here to the ffmpeg issue report once that is created.

[basisbit]

we don't use ffplay, but instead directly access the ffmpeg apis. There is no easy "-vcodec mpeg4_mmal" for that API as far as I know. The workaround which you mentioned would be very version dependent because of how often the ffmpeg API changes.

[manureini]

There is a avcodec_find_encoder function, but idk something about the ffmpeg api. I'll try to remove some encoders and decoders from ffmpeg with configure parameters, before I report that problem. ffmpeg doesn't show the mmal hardware acceleration on the ffmpeg -hwaccels list. But that's a know problem. Maybe because the pi is a special thing idk.

[manureini]

OK I tried to find out why the sound wasn't working with ffmpeg 3.2. The new commits (0aaba49f99104d53cb213f116459a88f271fb189) fixed the problem. I thought the problem is my linux sound configuration. Finally I can play 720p videos smoothly. My ffmpeg 3.2 configuration:--arch=arm --enable-pthreads --enable-runtime-cpudetect --prefix=/usr --enable-neon --enable-bzlib --enable-libfreetype --enable-gpl --shlibdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=armv7-a --extra-cflags='-mfpu=neon -fPIC -DPIC' --enable-shared --disable-static --enable-pic --enable-libx264 --enable-mmal --enable-hwaccel=h264_mmal --enable-omx-rpi --enable-omx --enable-opengl --enable-nonfree --enable-vdpau --enable-vaapi --enable-libtheora --enable-decoder=h264_mmal --enable-decoder=mpeg2_mmal --enable-decoder=mpeg4_mmal --enable-encoder=h264_omx --enable-encoder=h264_omx --enable-avresample --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libopus --disable-decoder=h264 --disable-decoder=mpeg4 --disable-encoder=libx264 --disable-encoder=mpeg4

Thanks for your help!

[chicoraf]

@lordmampf , I'm trying to install USDX on Raspberry Pi 3, but the complilation of fpc is failing. Can you share your compiled USDX, or point me to another tutorial (I'm following http://forum.ultra-star.de/viewtopic.php?f=13&t=11234 )?

Thanks a lot!

[manureini]

@chicoraf I can't remember if a made any changes with the fpc compiling stuff. But I wrote my commands down: check this out: https://gist.github.com/lordmampf/f4e3110539c4944d2349b0946b95cd5e Please tell me whether you have success.

[chicoraf]

@lordmampf , I was trying with fpc 3.0.2. With version 3.0.0 it worked. One more question: how is the performance? Here I'm getting 8 FPS in the menu, with 90% cpu utilization. In a music with video, the ftp drops to 4, turning the game unplayable. I have a Raspberry pi3 Model B. Any advice to improve performance?

Thanks for the help!

[basisbit]

advice: enable the experimental OpenGL driver for the raspberry Pi.

[alopezbu]

@basisbit I activate the experimental OpenGL drive and the sound of the program goes off. Nothing does sound, neither the songs nor the microphone

[joseluratm]

@basisbit I activate the experimental OpenGL drive and the sound of the program goes off. Nothing does sound, neither the songs nor the microphone

did you found a solution for the sound problem? thanks!

[cRaZy-bisCuiT]

Has anyone had success getting USDX working well on RPI4 with 720p or even 1080p?

[hongkongkiwi]

I would like to know this also!

[cRaZy-bisCuiT]

I'm working on this topic again. I'm building a Docker Image that will create RPI2,3,4 builds for Raspian (Debian 10 Buster). If everything works out I'll provide the source of the Dockerfile here as well as build instructions on the Wiki.

[s09bQ5]

For RPi 3 & 4 we could easily provide an ARM64 Flatpak built on Travis. But we probably need GL4ES or a custom build of Mesa to make it run.

[basisbit]

afaik GL -> GLES translation is already part of default raspbian nowadays for quite some time.

[s09bQ5]

You can configure Raspbian somehow to provide non-ES OpenGL, but AFAIK it is not the default. In a Flatpak we are limited to the libraries and DRI drivers provided in the freedesktop.org base platform. If there is no OpenGL Mesa build, we need to ship our own.

[basisbit]

I just checked. It is part of raspbian, and the user only has to enable it using raspi-config. So, no need to provide it in the package.

[s09bQ5]

Using GL4ES in the Flatpak would also be useful for other devices. Actually I was planning to run USDX on a cheap S905W device, but the audio drivers are still in development... Oh wait! They updated the website and it looks like audio drivers have landed in kernel v5.7!

[basisbit]

from my experience, the bigger problem is video playback on raspberry pi in usdx. It is quite hard to get ffmepg to use the decoding hardware support for h264.

[s09bQ5]

Is it? Hardware decoding on RPi is supported by the mmal and omx drivers in libavcodec.

Edit 1: This forum post says MMAL should be preferred over OpenMAX IL.

Edit 2: Ok, the MMAL driver in ffmpeg is decode-only and the OpenMAX IL driver encode-only.

[cRaZy-bisCuiT]

I did not get mmal compiled in ffmpeg, there seem to be some sources missing. I tried to remove any h.264 software decoding capabilities from ffmpeg to force it to use mmal - if I get the later added.

Is there a armv8/x64 raspian build that works with RPi4? Otherwise a ARM64 Flatpak built on Travis will not work.

[s09bQ5]

The mmal headers are in https://github.com/raspberrypi/userland and there is an aarch64 Raspbian since yesterday. I also received a 3A+ yesterday and intend to play with it this weekend.

[s09bQ5]

Playing USDX on a 512MB Raspberry Pi is no fun. Even after disabling almost all application and services that are automatically started in Raspbian, you immediately hit the point where it starts to swap to keep its 256MB CMA area. Trying to use the h264_mmal codec in that situation led to AVERROR_UNKNOWN. The h264_v4l2m2m codec didn't return an error but appeared to swallow so much RAM that the OpenGL graphics no longer worked.

Decoding videos in software does work but of course stutters for high resolution videos.

ProjectM crashes USDX until you fix the path to the fonts in /usr/share/projectM/config.inp. That problem appears to exist on all Debian platforms (Debian bug 961971)

I could reproduce the problem with linking mentioned in #525 . Debian relies on fpcmkcfg to put the path to GCC in /etc/fpc.cfg. But that tool will do that only on x86 and powerpc (FPC bug 37158).

[cRaZy-bisCuiT]

Maybe we should file a bug report for fpcmkcfg. At least it's a missing feature I'd say.

[daniel-j]

Nice to see more work on improving RPi support. On my Pi 4 i disable video playback and the game runs fine :)

[cRaZy-bisCuiT]

Nice to see more work on improving RPi support. On my Pi 4 i disable video playback and the game runs fine :)

I know. But my intention is to make 1080p h.264 without stutter at least at PI4@4GB RAM possible as well. I'm not sure if the driver stack on armhf and/or arm64 + USDX/Pascal do make that possible. USDX is a great project and game, but the resource usage is abysmal at times. Maybe still some work at the driver stack is needed but I can only say that for sure when I have time to make some progress on my build.

[s09bQ5]

Maybe we should file a bug report for fpcmkcfg.

I did and the bug report is linked above. No reaction from the Free Pascal people so far.

Have you tried adding code to USDX to select the h264_mmal codec instead of the default h264 codec? The libavcodec shipped with Raspbian includes that codec.

I envisioned a solution where codec priorities could be set in config.ini, but then my elegant implementation changed the codec order for all formats because TList.QuickSort is not a stable sorting function the way they implemented it in Free Pascal.

[cRaZy-bisCuiT]

I remember basisbit mentioned ffmpeg is supposed to use the best decoding option it has to do decoding. I'm note sure if there's a way of selecting h264_mmal when running on any kind of ARM arch.

I guess I don't have so much time to get into the source and pascal that much right now. Would you have any ideas how to implement that?

Setting that either while compiling or in config.ini / as a parameter when launching would be amazing.

My approach was to disable all kinds of software decoding via build flag for h264 to hope ffmpeg will make use of mmal then.

In addition, I saw libvlc is also a dependency: So is ffmpeg still being used anyway? What is libvl for?

[s09bQ5]

But the fact is that ffmpeg does not try to select the best codec. It has a list of all codecs and avcodec_find_decoder selects the first codec from that list that claims to be able to decode the format. The ffmpeg people deliberately put software codecs before hardware codecs. See f.ex. https://github.com/FFmpeg/FFmpeg/commit/7c27da686c06d219258270643088d2f9b393b585 .

Applications that want to use a specific codec call avcodec_find_decoder_by_name.

We don't use libvlc. Where did you see that?

[s09bQ5]

Maybe the easiest solution is to allow users to simply list those codecs that should be preferred. On startup we resolve these names to AVCodec pointers and each time we want to open a file, we go through that list of pointers and try each codec that has a matching AVCodecID before falling back to avcodec_find_decoder. To help users create that list, one could log the names of all video decoders if an invalid name is found in the list.

I think I'll implement this when I get back from work.

[cRaZy-bisCuiT]

Sounds great, looking forward to your commit to learn something new. :) This would be very nice for Linux x86 guests as well since the performance could also be not that good depending on the hardware.

With your solution one might use the exactly same infrastructure which is used to build DEB x86 packages to build for arm64 platform with Travis CI. That would be great! In this case only some kind of cross-compiler toolchain would be needed for armhf builds. Lowering the complexity should make the whole project maintainable in a more convinient way.

In addition: How did you monitor which codec ffmpeg uses to decode h.264? Is there a log written somewhere where you could find out about that? Did you compile USDX with some debug flags?

[s09bQ5]

See PR #530.

I have an x86 ION 330 board that is unable to decode full HD H.264 @ 30 FPS on the CPU but can easily do this with VDPAU. Unfortunately VDPAU is one of those APIs that need av_hwframe_transfer_data (and ideally one would use the NV_vdpau_interop extension instead of copying the frame to system memory).

I have added a Log.LogInfo line that prints the name of the used codec to Error.log.