search

caprica / picam

Unofficial Java API library for the Raspberry Pi camera.
GNU General Public License v3.0
49 stars 11 forks source link

Native image encoder intermittently fails to send final frame of image data causing hang #2

Closed caprica closed 5 years ago

caprica commented 7 years ago

From time to time, when requesting an image, the native image encoder simply fails to send the final frame of image data.

The image data is fetched in chunks, this is repeated 1..n times by the native code invoking a callback with the image data. The return value of that callback signifies to the native encoder that it should send the next frame. Tracing the Java code shows that the callback is correctly invoked, and that the correct return value is being sent back to native code. Nevertheless, from time to time the native encoder simply seems to stall and will stop sending image data. It always seems to be the last chunk of data.

What this means in practical terms is that the Java application hangs while waiting for a native callback that will never occur. The JVM must then be terminated.

This is a show-stopper issue that precludes the use of this library.

There is similar strangeness, it may be unrelated, during native resource clean-up whereby native warning messages are intermittently dumped to stdout or stderr (e.g. freeing a resource that has already been freed).

This may point to there being a fundamental problem with the implementation of this library.

Having said that, the code for handling capture data from the native library has been implemented and debugged very carefully from the ground up on two entirely separate occasions. So either there is something wrong in native code that the JVM is exposing, or the exact same mistake has been made on two separate Java implementations.

Until the root cause of this issue can be identified, this library can not be used reliably.

Any hints from anyone and anywhere on what could be wrong here will be most welcome.

AlexanderBykin commented 7 years ago

hi, found your beautiful implementation, here is RaspiMJPEG project written with C language, this utility takes snapshots as fast as possible and flush them into the disk, i think this can help you a lot

caprica commented 7 years ago

Thanks!

The Java implementation here is based on the C code for the RaspiStill utility in the project that you linked to.

Nevertheless I will look at the particular code you linked and see if I can spot anything I did wrong.

AlexanderBykin commented 7 years ago

here is also parts of RaspiMJPEG: RaspiMCmds RaspiMJPEG this utility run in a while loop and opens FIFO there you can send commams to control camera things

AlexanderBykin commented 7 years ago

yes you're right, this utility based on RaspiStill, but author of this utility has increased speed of taking pictures

caprica commented 7 years ago

That code does not really help too much, the key idea behind this Java implementation is to use the MMAL API directly and not rely on forking processes and using native commands.

Some more background on this issue...

I very carefully did two separate Java implementations of this, from scratch, based on the source code in the raspicam project, in particular the source code that used the native MMAL API directly. Both of those implementations I did suffered the exact same problems, so clearly I missed something non-obvious, twice.

It is very much possible to invoke the native command-line tools from Java, and there are already Java projects that do that. But I do not want to do that, I want a Java API that does not execute native commands in their own processes, but instead to implement the exact same thing that those native commands do when they talk to the MMAL API.

This is much more difficult, and requires me to write a lot more code, but the end result in my opinion will be much better. If it only worked 100% reliably!

AlexanderBykin commented 7 years ago

yes, exactly for direct calls to MMAL API from Java by using JNA i liked your project

caprica commented 7 years ago

I still hope to find a resolution to this issue, so the project is still very much alive, it's just stalled right now.

AlexanderBykin commented 7 years ago

maybe this is a bug of MMAL API? try to update Raspberry rpi-update and check there is the fresher VC libraries at /opt/vc

AlexanderBykin commented 7 years ago

also take me a piont where is that happening at code, i want to see how can i help you

caprica commented 7 years ago

I can't really point directly to the code that is broken.

The code dealing with handling picture data is in https://github.com/caprica/picam/blob/master/src/main/java/uk/co/caprica/picam/EncoderBufferCallback.java

This is a Java implementation of a native callback invoked by the MMAL API when camera data is ready to be sent. This callback accepts the picture data and then either finishes if it's the last data block, or requests more picture data if it is not the last data block. The problem is that this code executes correctly, it requests the next block when needed, but sometimes the native code simply refuses to invoke this callback with the last black of data. This failure happens outside of Java code.

Now, it could be that the Java code has exposed a rare bug in the native code, or it could be that there is something wrong in the Java code. Another problem is that there is a lot of stuff that has to be set up and linked together via the MMAL API, and any part of that could be slightly wrong.

All of the native setup code is here https://github.com/caprica/picam/blob/master/src/main/java/uk/co/caprica/picam/Camera.java

caprica commented 7 years ago

Your post inspired me to fully upgrade my Pi and after doing so I took 50 pictures without any problems - no hanging.

I'll do some more testing.

I can't say this issue is definitely gone, but so far it's working better than before.

AlexanderBykin commented 7 years ago

it's still here, i get it time to time

AlexanderBykin commented 7 years ago

can you say how did you test it? you just running several times from terminal?

caprica commented 7 years ago

Oh well, my testing was just ad-hoc casual testing using my own test application. Doing that same test previously it failed about 10% of the time. Today it hasn't failed yet.

I was going to write a proper stress-test later.

I guess there's still an issue.

AlexanderBykin commented 7 years ago

i have added extra logs

11:51:01.738 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [apply] apply()
11:51:01.738 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [apply] flags=0
11:51:01.739 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [apply] bufferLength=0
11:51:01.740 [Thread-35] INFO uk.co.caprica.picam.EncoderBufferapply - >>> [apply] BufferedLength=0
11:51:01.741 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [sendNextPictureBuffer] sendNextPictureBuffer()
11:51:01.746 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [sendNextPictureBuffer] result=0
11:51:01.747 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [apply] finished=false
11:51:01.747 [Thread-35] DEBUG uk.co.caprica.picam.EncoderBufferapply - [apply] enabled=true

looks like buffer lost his data somewhere

caprica commented 7 years ago

That is indeed the same behaviour I was seeing previously.

And yeah, that's a native buffer, not a Java buffer.

caprica commented 7 years ago

Sadly my stress test did not show good results. Strange messages from MMAL dumped to the error console, looks like a native race to me. Still hangs intermittently.

So updating everything to latest does not resolve the issue, unfortunately. I still don't know what's actually wrong.

caprica commented 7 years ago

This old abandoned issue looks somewhat similar:

https://www.raspberrypi.org/forums/viewtopic.php?t=108707&p=749409

caprica commented 7 years ago

With debug logging turned on on the Pi, I occasionally see these warnings in the log (they do not correspond to a hang):

599057.418: gpioman: gpioman_get_pin_num: pin CAMERA_LED not defined
600910.823: mmal: mmal_port_event_send: event lost on port 1,0 (buffer header callback not defined)
600937.237: gpioman: gpioman_get_pin_num: pin CAMERA_LED not defined
604524.678: mmal: mmal_port_disable: port ril.image_encode:out:0(PNG )(3e6df080) is not enabled
604524.702: mmalsrv: mmal_do_port_disable: could not disable port (3:EINVAL)
604535.891: mmal: mmal_port_disable: port ril.image_encode:out:0(PNG )(3e6df080) is not enabled
604535.918: mmalsrv: mmal_do_port_disable: could not disable port (3:EINVAL)
604600.604: mmal: mmal_port_disable: port ril.image_encode:out:0(PNG )(3e6df080) is not enabled
604600.630: mmalsrv: mmal_do_port_disable: could not disable port (3:EINVAL)

When I CTRL+C a hanged app, this line pops up in the log:

775057.940: mmalsrv: send_buffer_to_host: tx failed:size 292 st -1

My stress-test app ran to 90 out of 100 pictures taken.

What's interesting is it runs better with more logging (so it seems) and that to me implies a race somewhere.

caprica commented 7 years ago

This also looks somewhat related:

https://github.com/raspberrypi/firmware/issues/417

AlexanderBykin commented 7 years ago

with this part of code it looks like nothing happens my repo

public class MainApp {
    public static void main(String[] agrs) {
        bcm.bcm_host_init(); // FIXME seems not needed?

        CameraConfiguration config = CameraConfiguration.cameraConfiguration()
                .width(1920)
                .height(1080);

        try {
            Camera camera = new Camera(config);
            camera.takePicture(new FilePictureCaptureHandler(new File("/dev/shm/picam.png")));
        } catch (Exception ex) {
            ex.printStackTrace();
        }
        bcm.bcm_host_deinit(); // FIXME seems not needed?
    }
}
AlexanderBykin commented 7 years ago

ah just got it again, never mind :) i just thought that is related to bcm_host initialization

caprica commented 7 years ago

Yeah, I've been down that path myself before.

The reason it's there is because it was in the original C code.

caprica commented 7 years ago

I'm afraid I don't really know the MMAL API, nor the Pi platform. I just attempted to port the code from C to Java/JNA.

AlexanderBykin commented 7 years ago

i am in the same way :)

AlexanderBykin commented 7 years ago

snapshots also possible by DispmanX it is part of Raspberry VideoCore

caprica commented 7 years ago

Ideally I'd like to find the minimal possible working native code to get a snapshot/still-picture. I don't care about video. I don't care what native API I use to get it.

AlexanderBykin commented 7 years ago

ok, then this is what you looking for DispmanX all of this methods is places at libbcm_host.so before i found your project i try to use it but i have got a problem, how to decode RAW data into PNG or Jpeg my code is ugly, i dont want to publish it

AlexanderBykin commented 7 years ago

and here is an example how to use DispmanX

AlexanderBykin commented 7 years ago

DispmanX also gives a possibility to place Overlay image

caprica commented 7 years ago

There must be a Java library to convert RAW to PNG.

I am going to try and remove the PNG encoding from the current implementation first. I have a suspicion the problem is there.

caprica commented 7 years ago

That dispman code is taking a snapshot of the raspberrypi screen, it's not using the camera??

AlexanderBykin commented 7 years ago

it is taking snapshot based on Display number, just make sure you're used displayNumber = 0 to use camera

caprica commented 7 years ago

OK, it wasn't clear, thanks.

AlexanderBykin commented 7 years ago

i might be wrong, corrent me if i am wrong. there is good book about Raspberry GPU programming: Raspberry Pi GPU Audio Video Programming written by Jan Newmarch

caprica commented 7 years ago

Frankly, such a thing is outside the scope of this project. My intent was to port an existing native application line-by-line to Java and put a nice API around it, and that's it. I find the native application code I've seen so far to be overly complicated to be honest with you, and there's a distinct lack of concise example code for MMAL as far as I have found.

caprica commented 7 years ago

So with some refactoring, whereby I setup and teardown more native resources once rather than for every picture, things worked a bit better.

But unfortunately, I got a hang after about 50 pictures on average (1-second delay per picture).

AlexanderBykin commented 7 years ago

looks like contributors of raspberry userland wanted to help us, will we help them to find the bug?

caprica commented 7 years ago

Kind of passive-aggressive though, huh.

This project is not my main concern right now, sorry.

caprica commented 7 years ago

First test run with all debug enabled resulted in no hang.

Second test run failed on third picture.

pi@raspberrypi:~/workspaces/test $ VC_LOGLEVEL="mmal:trace" java -jar picam.jar 250 > log.txt
mmal: mmal_port_alloc: component:vc.null_sink type:1 extra:0
mmal: mmal_port_alloc: vc.null_sink:ctr:0: created at 0x656ac6b0
mmal: mmal_vc_component_create: vc.null_sink
mmal: mmal_vc_component_create: vc.null_sink: handle 0x40 status 0 reply status 0
mmal: mmal_port_free: vc.null_sink:ctr:0 at 0x656ac6b0
mmal: mmal_port_alloc: component:vc.null_sink type:1 extra:384
mmal: mmal_port_alloc: vc.null_sink:ctr:0: created at 0x656acd40
mmal: mmal_port_alloc: component:vc.null_sink type:2 extra:384
mmal: mmal_port_alloc: vc.null_sink:in:0: created at 0x656ad050
mmal: mmal_vc_component_create:  handle 64
mmal: mmal_vc_port_info_get: get port info (1:0)
mmal: mmal_vc_port_info_get: get port info (2:0)
mmal: mmal_pool_create_with_allocator: allocating 48 + 24 + 472 * 4 bytes for pool
mmal: mmal_pool_initialise_buffer_headers: allocating 96 bytes for payload 0/4
mmal: mmal_pool_initialise_buffer_headers: allocating 96 bytes for payload 1/4
mmal: mmal_pool_initialise_buffer_headers: allocating 96 bytes for payload 2/4
mmal: mmal_pool_initialise_buffer_headers: allocating 96 bytes for payload 3/4
mmal: mmal_component_create_core: created 'vc.null_sink' 0 0x656ac4f0
mmal: mmal_component_enable: vc.null_sink 0
mmal: mmal_component_create: vc.ril.image_encode
mmal: mmal_port_alloc: component:vc.ril.image_encode type:1 extra:0
mmal: mmal_port_alloc: vc.ril.image_encode:ctr:0: created at 0x656b0500
mmal: mmal_vc_component_create: vc.ril.image_encode
mmal: mmal_vc_component_create: vc.ril.image_encode: handle 0x38 status 0 reply status 0
mmal: mmal_port_free: vc.ril.image_encode:ctr:0 at 0x656b0500
mmal: mmal_port_alloc: component:vc.ril.image_encode type:1 extra:384
mmal: mmal_port_alloc: vc.ril.image_encode:ctr:0: created at 0x656b08c0
mmal: mmal_port_alloc: component:vc.ril.image_encode type:2 extra:384
mmal: mmal_port_alloc: vc.ril.image_encode:in:0: created at 0x656b0bd0
mmal: mmal_port_alloc: component:vc.ril.image_encode type:3 extra:384
mmal: mmal_port_alloc: vc.ril.image_encode:out:0: created at 0x656b0ef0
mmal: mmal_vc_component_create:  handle 56
mmal: mmal_vc_port_info_get: get port info (1:0)
mmal: mmal_vc_port_info_get: get port info (2:0)
mmal: mmal_vc_port_info_get: get port info (3:0)
mmal: mmal_pool_create_with_allocator: allocating 48 + 24 + 472 * 4 bytes for pool
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 0/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 1/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 2/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 3/4
mmal: mmal_component_create_core: created 'vc.ril.image_encode' 1 0x656b0330
mmal: mmal_port_format_commit: vc.ril.image_encode(3:0) port 0x656b0ef0 format 3:PNG 
mmal: mmal_vc_port_info_set: set port info (3:0)
mmal: mmal_vc_port_info_get: get port info (3:0)
mmal: mmal_component_enable: vc.ril.image_encode 1
mmal: mmal_port_pool_create: vc.ril.image_encode(3:0) port 0x656b0ef0, headers 1, size 81920
mmal: mmal_pool_create_with_allocator: allocating 48 + 8 + 472 * 1 bytes for pool
mmal: mmal_pool_initialise_buffer_headers: allocating 81920 bytes for payload 0/1
mmal: mmal_port_payload_alloc: vc.ril.image_encode(3:0) port 0x656b0ef0, size 81920
mmal: mmal_vc_port_payload_alloc: vc.ril.image_encode:out:0(PNG ): allocating 81920 bytes, format PNG , is_zero_copy 0
mmal: mmal_vc_port_payload_alloc: vc.ril.image_encode:out:0(PNG ): allocated at 0x656b70f0
mmal: mmal_port_acquire: port vc.ril.image_encode:out:0(PNG )(0x656b0ef0), refcount 0
mmal: mmal_component_create: vc.ril.camera
mmal: mmal_port_alloc: component:vc.ril.camera type:1 extra:0
mmal: mmal_port_alloc: vc.ril.camera:ctr:0: created at 0x656b4f20
mmal: mmal_vc_component_create: vc.ril.camera
mmal: mmal_vc_component_create: vc.ril.camera: handle 0x30 status 0 reply status 0
mmal: mmal_port_free: vc.ril.camera:ctr:0 at 0x656b4f20
mmal: mmal_port_alloc: component:vc.ril.camera type:1 extra:384
mmal: mmal_port_alloc: vc.ril.camera:ctr:0: created at 0x656b52e0
mmal: mmal_port_alloc: component:vc.ril.camera type:3 extra:384
mmal: mmal_port_alloc: vc.ril.camera:out:0: created at 0x656b55f0
mmal: mmal_port_alloc: component:vc.ril.camera type:3 extra:384
mmal: mmal_port_alloc: vc.ril.camera:out:0: created at 0x656b5900
mmal: mmal_port_alloc: component:vc.ril.camera type:3 extra:384
mmal: mmal_port_alloc: vc.ril.camera:out:0: created at 0x656b5c10
mmal: mmal_port_alloc: component:vc.ril.camera type:4 extra:384
mmal: mmal_port_alloc: vc.ril.camera:clk:0: created at 0x656b5f20
mmal: mmal_vc_component_create:  handle 48
mmal: mmal_vc_port_info_get: get port info (1:0)
mmal: mmal_vc_port_info_get: get port info (3:0)
mmal: mmal_vc_port_info_get: get port info (3:1)
mmal: mmal_vc_port_info_get: get port info (3:2)
mmal: mmal_vc_port_info_get: get port info (4:0)
mmal: mmal_pool_create_with_allocator: allocating 48 + 24 + 472 * 4 bytes for pool
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 0/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 1/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 2/4
mmal: mmal_pool_initialise_buffer_headers: allocating 740 bytes for payload 3/4
mmal: mmal_component_create_core: created 'vc.ril.camera' 2 0x656b4d50
mmal: mmal_port_parameter_set: vc.ril.camera(1:0) port 0x656b52e0, param 0x656b6f60 (10010,12)
mmal: mmal_port_parameter_set: vc.ril.camera(1:0) port 0x656b52e0, param 0x656cc0e8 (10037,12)
mmal: mmal_port_enable: vc.ril.camera:ctr:0 port 0x656b52e0, cb 0x7630f008, buffers (4/0/4,740/0/740)
mmal: mmal_port_parameter_set: vc.ril.camera(1:0) port 0x656b52e0, param 0x656b4120 (10015,48)
mmal: mmal_port_parameter_set: vc.ril.camera(1:0) port 0x656b52e0, param 0x656ce3d8 (10025,24)
mmal: mmal_port_format_commit: vc.ril.camera(3:2) port 0x656b5c10 format 3:OPQV
mmal: mmal_vc_port_info_set: set port info (3:2)
mmal: mmal_vc_port_info_get: get port info (3:2)
mmal: mmal_component_enable: vc.ril.camera 2
mmal: mmal_connection_create: out 0x656b55f0, in 0x656ad050, flags 3, vc.ril.camera:out:0/vc.null_sink:in:0
mmal: mmal_port_format_commit: vc.null_sink(2:0) port 0x656ad050 format 3:I420
mmal: mmal_vc_port_info_set: set port info (2:0)
mmal: mmal_vc_port_info_get: get port info (2:0)
mmal: mmal_port_connect: connecting vc.ril.camera:out:0(0x656b55f0) to vc.null_sink:in:0(I420)(0x656ad050)
mmal: mmal_connection_enable: 0x656ce520, vc.ril.camera:out:0/vc.null_sink:in:0
mmal: mmal_port_enable: vc.ril.camera:out:0 port 0x656b55f0, cb (nil), buffers (1/1/1,7680/7680/7680)
mmal: mmal_vc_port_info_get: get port info (3:0)
mmal: mmal_connection_create: out 0x656b5c10, in 0x656b0bd0, flags 3, vc.ril.camera:out:2/vc.ril.image_encode:in:0
mmal: mmal_port_format_commit: vc.ril.image_encode(2:0) port 0x656b0bd0 format 3:OPQV
mmal: mmal_vc_port_info_set: set port info (2:0)
mmal: mmal_vc_port_info_get: get port info (2:0)
mmal: mmal_vc_port_info_get: get port info (3:0)
mmal: mmal_port_connect: connecting vc.ril.camera:out:2(OPQV)(0x656b5c10) to vc.ril.image_encode:in:0(OPQV)(0x656b0bd0)
mmal: mmal_connection_enable: 0x656ce600, vc.ril.camera:out:2/vc.ril.image_encode:in:0
mmal: mmal_port_enable: vc.ril.camera:out:2(OPQV) port 0x656b5c10, cb (nil), buffers (10/10/3,128/128/128)
mmal: mmal_vc_port_info_get: get port info (3:2)
mmal: mmal_port_enable: vc.ril.image_encode:out:0(PNG ) port 0x656b0ef0, cb 0x7630f030, buffers (1/1/1,81920/81920/81920)
# take picture 1/250
mmal: mmal_port_parameter_set: vc.ril.camera(3:2) port 0x656b5c10, param 0x656a2290 (10011,12)
# take picture 2/250
mmal: mmal_port_parameter_set: vc.ril.camera(3:2) port 0x656b5c10, param 0x656b4460 (10011,12)
# take picture 3/250
mmal: mmal_port_parameter_set: vc.ril.camera(3:2) port 0x656b5c10, param 0x656b4460 (10011,12)
# hang
AlexanderBykin commented 7 years ago

the same for me mmal: mmal_port_parameter_set: vc.ril.camera(3:2) port ... and nothing happens

caprica commented 7 years ago

That port parameter set triggers the take picture, the hang is not there, it is later.

AlexanderBykin commented 7 years ago

for me it is stop working after mmal_port_parameter_set

caprica commented 7 years ago

The mmal_port_parameter_set is the last thing you see in that particular trace log, but that is not where the hang is.

AlexanderBykin commented 7 years ago

just checked one thing, modified picam code to initialize once and take 100 pictures continuously, all pictures taken successfully. it looks like something wrong with MMAL initialization.

caprica commented 7 years ago

I already have a local change that does the same thing, I mentioned it above. It works better, but is still not 100%. I agree there is likely something wrong with MMAL initialisation, there is a lot of "stuff" to link together it seems, and I may well have got something wrong. I still don't know what though.

vrosca commented 7 years ago

Hi there,

After having spent a full day fiddling around with the Java-PICamera integration, I'm happy to share a solution to the hanging problem. What kind of gave it away was the fact that with JPEG encoding (hardware-accelerated, only 100ms for full consumption via raspistill), picam always only manages to get a filled buffer on the first callback. The subsequent callbacks come with 0 buffer length and it hangs on the last one just like the PNG case. T

Solution: use Pointer + .read() instead of the _T structures for the callback interface, JNA messes things up somewhere in the translation layer.

public interface MMAL_PORT_BH_CB_T extends Callback {

void apply(Pointer port, Pointer buffer);

}; ... int mmal_port_send_buffer(Pointer portPointer, Pointer buffer);

int mmal_buffer_header_mem_lock(Pointer header);

void mmal_buffer_header_mem_unlock(Pointer header);

void mmal_buffer_header_release(Pointer header);

I was looking for higher performance and made the following adjustments as well:

  1. Moved towards direct access by making a LibFastMmal interface with static natives and Native.register(LIBRARY_NAME);

  2. Used a single thread for callbacks by registering the callback with a CallbackThreadInitializer

private final CallbackThreadInitializer callbackThreadInitializer = new CallbackThreadInitializer(true, false, "MMALCallback"); // in Camera .. Native.setCallbackThreadInitializer(encoderBufferCallback, callbackThreadInitializer); // in takePicture .. callbackThreadInitializer.detach(encoderBufferCallback); // in takePicture

This enables a full-frame JPEG (produced by camera v2 at 3280 x 2464) to be consumed by the callback in 0.5 seconds at the first invocation and 0.25 seconds after the JVM heats up (on a pi zero w with oracle java 1.8.0_144-b01).

Tested the pointer fix in a loop with 1000 java cold-starts and it hasn't hanged so far.

Some stringent performance requirements for my application might require me to move away from this approach to only one callback with the full buffer via JNI (couldn't get the time below 250ms while raspistill manages to do the same thing in 100ms). Nevertheless, hope the above helps and thanks for sharing the picam code.

BR --Val

caprica commented 7 years ago

Wow. I need to digest this more thoroughly, but thank you very much.

vrosca commented 7 years ago

You're welcome, I'd share the code if I hadn't butchered it to an unrecognisable extent.

To try out the main fix you should just change the signatures in LibMmal / MMAL_PORT_BH_CB_T / EncoderBufferCallback / CamercaControlCallback as above and you're almost set. Then in EncoderBufferCallback.appply() you can easily dereference the pointers as such:

public void apply(Pointer portPointer, Pointer bufferPointer) { ... MMAL_BUFFER_HEADER_T buffer = new MMAL_BUFFER_HEADER_T(bufferPointer); buffer.read(); ... mmal_buffer_header_mem_lock(bufferPointer) ... mmal_buffer_header_mem_unlock(bufferPointer); ... mmal_buffer_header_release(bufferPointer); ... sendNextPictureBuffer(portPointer);

mikelduke commented 7 years ago

I made the changes suggested by vrosca, but must have missed something. I can capture more images, but it still hangs after a while.