Native image encoder intermittently fails to send final frame of image data causing hang

[caprica]

From time to time, when requesting an image, the native image encoder simply fails to send the final frame of image data.

The image data is fetched in chunks, this is repeated 1..n times by the native code invoking a callback with the image data. The return value of that callback signifies to the native encoder that it should send the next frame. Tracing the Java code shows that the callback is correctly invoked, and that the correct return value is being sent back to native code. Nevertheless, from time to time the native encoder simply seems to stall and will stop sending image data. It always seems to be the last chunk of data.

What this means in practical terms is that the Java application hangs while waiting for a native callback that will never occur. The JVM must then be terminated.

This is a show-stopper issue that precludes the use of this library.

There is similar strangeness, it may be unrelated, during native resource clean-up whereby native warning messages are intermittently dumped to stdout or stderr (e.g. freeing a resource that has already been freed).

This may point to there being a fundamental problem with the implementation of this library.

Having said that, the code for handling capture data from the native library has been implemented and debugged very carefully from the ground up on two entirely separate occasions. So either there is something wrong in native code that the JVM is exposing, or the exact same mistake has been made on two separate Java implementations.

Until the root cause of this issue can be identified, this library can not be used reliably.

Any hints from anyone and anywhere on what could be wrong here will be most welcome.

[vrosca]

@mikelduke please provide the exact code you use to run the whole thing (and if you can - specify where it hangs for you, with a thread dump) and I'd be happy to take a look. I'm running this on a single core Pi Zero - let me know if you use a different Pi as it could behave differently on a multicore (even though the underlying openmax code does seem to always callback on the same thread as far as I can tell from the pthread id).

PS: pulled your repo and my basic JPEG test ran well with your code, while it always hanged on the base version - that's why I'd like a more specific hanging scenario

[AlexanderBykin]

@vrosca can you upload your changes? we all together can test it

[vrosca]

For my particular scenario I moved towards a mixed JNI/JNA approach that builds around MMAL in a different way than picam, that's why I have no changes to share. Anyways, the changeset by @mikelduke in the repo linked above seems to encompass my feedback well. Once I can reproduce the exact hang he sees I'll happily upload a solution ... assuming I have one

[caprica]

For problems like this, I just have this nagging doubt it's a classic race and that what these changes did was make the whole thing more efficient so the issue is seen less often, but an underlying fundamental issue still remains.

But that's just my feeling. I haven't dug into these changes in any detail.

[vrosca]

I thought so too initially, however the fact that the callback thread always yields the same pthread id (within the same established connection) suggests there are no other threads at play that can deadlock. I suspect the issues @mikelduke has actually come not from the data callback but the fact enabling/disabling the components in a fast loop DOES lead to strange jams. Checking the Python picamera library or even the raspistill source code I haven't seen attempts for a fast succession of complete disable/enable sequences. Might be best to refactor takePicture if the problem occurs at a deeper layer. Anyways, I'll wait for the test code leading to a hung state before speculating further.

[mikelduke]

@vrosca I don't have access to my pi for the next few days to test it again, but I was running on RPi2b so it is multi-core. It would take about 30 pics before hanging in the Camera.destroyComponent(cameraComponent); method.

[vrosca]

Good, that validates my hunch, it hangs on rapid destruction/creation of the camera component, not the take picture part we amended. I'm guessing you wrapped the exiting sample code in one big for loop and kept the default delay? Because then you'd end up destroying/creating the camera component with a very small delay - getting it to eventually jam. Managed to get raspistill to jam too with this approach, the issue might be deeper - nevertheless this is an exotic use case, who needs to destroy the camera every time they take a picture?

Find attached a patch containing a JPEG test, runs reliably for me. jpegTestAndDelayFix.zip This includes a trivial fix for the delay=0 configuration (where pictureData gets called before begin sometimes).

@caprica if the RPI2b will also work without issues, I think you can consider merging those changes back in

[caprica]

For what it's worth, I did some of my own refactoring a while back so that it did not create/destroy the camera component for each picture. I still saw hangs.

My original idea was to get create -> take picture -> destroy working 100% reliably, and use that as a base to get something better later, but that didn't work of course.

[vrosca]

There are two separate problems: 1) the pictureData callback working reliably 2) a rapid succession of camera/encoder destory/creates not blocking

I haven't invested any time into #2 - seems like a fairly exotic scenario. However, I'm starting to be reasonably confident that #1 is properly addressed now. Lets see what the RPI2b test yields, if it handles a multicore PI properly then it should be good to go

[mikelduke]

@vrosca Your hunch is spot on, that's exactly what I did. I also saw that once thread hangs, nothing else can access the camera without a reboot, even raspistill.

On Oct 17, 2017 9:59 AM, "vrosca" notifications@github.com wrote:

There are two separate problems:

1. the pictureData callback working reliably
2. a rapid succession of camera/encoder destory/creates not blocking

I haven't invested any time into #2 https://github.com/caprica/picam/issues/2 - seems like a fairly exotic scenario. However, I'm starting to be reasonably confident that #1 https://github.com/caprica/picam/issues/1 is properly addressed now. Lets see what the RPI2b test yields, if it handles a multicore PI properly then it should be good to go

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caprica/picam/issues/2#issuecomment-337258331, or mute the thread https://github.com/notifications/unsubscribe-auth/AFoIk3OztSgX094lI1hMPkI0gkeQdOMjks5stMDhgaJpZM4NeUZ0 .

[caprica]

Yeah, to be honest I don't really care if we never solve your scenario #2, that approach was never intended to be the end-game.

[mikelduke]

That's understandable, maybe just adding a note saying not to do that would be good.

On Oct 17, 2017 10:46 AM, "Mark Lee" notifications@github.com wrote:

Yeah, to be honest I don't really care if we never solve your scenario #2 https://github.com/caprica/picam/issues/2, that approach was never intended to be the end-game.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/caprica/picam/issues/2#issuecomment-337270796, or mute the thread https://github.com/notifications/unsubscribe-auth/AFoIk6okXCxva-EncdfDDhzsfOGSYPtWks5stMvIgaJpZM4NeUZ0 .

[techyian]

Trying to digest everything that's been mentioned in this thread already, but during the initial development of MMALSharp I used to see MMAL hangs due to not clearing port pools correctly. The hang was because there was still a reference on the buffer and wasn't released by MMAL.

So in MMALSharp specifically, I've managed to get a clean shutdown of resources by doing the following:

1) Once you've finished processing image data. Stop the output port(s) of your last connected component 2) Close connections of your connected components 3) Clean the port pools by calling mmal_port_pool_destroy - I can see this call in your code but it may be the timing that's slightly out. Also note, I've done this on input & output ports on all components, probably overkill but done just in case.

Happy to help if you need it. This problem was bugging me for months!

[caprica]

Thanks for such a helpful comment.

I may revisit this. I did try to be very careful about allocating and freeing resources, but I just couldn't see where I may have went wrong... on three separate, from scratch, implementations.

[caprica]

@techyian one follow-up question if you don't mind, in your clean-up suggestions, are you doing the create/process-image-data/cleanup for every image capture individually, or do you keep the camera "open" and do multiple captures before cleaning up?

[techyian]

Sure - if you take a look at the BeginProcessing method, I keep the camera component resources active, but cleanup ports and stop capture after each run. That way, people using the library can then add/remove components and create a different pipeline after each run if they wish to.

[vrosca]

Just wanted to mention that after taking the approach explained in my first post (23 Sep 2017) my surveillance camera now has 6 months of uptime without any freezes

[caprica]

That is impressive, thanks for the update.

Would be so awesome if you could isolate the essential code changes, last time you said the code was "butchered" :-D

[vrosca]

I'll try - at some point in Q2 - to get over my anti-git bias, rebase my changes on your codebase and send a pull request

[AlexanderBykin]

@vrosca just share your code as a zip file if you not good with Git

[caprica]

I finally found some time to revisit this.

I applied the changes suggested by @vrosca on 23rd September and so far I've been able to capture hundreds upon hundreds of pictures without any problems (my code uses the PNG encoder rather than JPG for what it's worth). My last test ran 1,000 straight captures perfectly.

I will work on cleaning up my code and committing it soon and maybe even pushing a release to Maven Central.

I still don't see anything "wrong" with the original code and I still worry about races, but so far these changes look good.

Thank you all in this thread who helped, especially @vrosca for the suggested code changes.

[AlexanderBykin]

Cool, do you have statistics on how many pictures it can take in one second?

[AlexanderBykin]

with C++ implementation it can take about 30 pictures per second and save them into Ramdisk

[caprica]

Honestly I have no idea since I have not measured it, but it seems slow-ish to me - then again, my tests so far save all images to disk rather than ramdisk so that's going to slow it down, and using the JPG encoder would likely improve things.

On the other hand, raw performance is not the primary goal of this project, rather the goal is ease of use and Java is a lot easier to deal with than C++, especially for less experienced developers.

[AlexanderBykin]

@caprica try to use /dev/shm/ this is ramdisk at Raspberry

[caprica]

Sure, but like I mentioned raw performance is not the primary goal of this project, although I would be interested in the numbers :)

To make any significant performance improvements likely would mean removing JNA and using JNI directly - and again, that will make things more difficult for people, not easier.

[AlexanderBykin]

I understand your position. when I found your project I thought it was a godsend for my project (controlled tank from a mobile phone) but when using it I was disappointed with extremely slow work tank picture

[caprica]

Well, now that it is hopefully working reliably, performance concerns could be looked at. But if you want best performance, use C/C++ rather than Java I suppose. I mean, the JVM is fundamentally slow on these low-power devices isn't it.

[AlexanderBykin]

I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try

[vrosca]

@Mark: glad the changes worked for you! I don't have the thread that fresh in my mind but seem to recall the issue was that JNA was automatically filling the structures (including buffer pointers) BEFORE you acquired the lock from the broadcom libs. When doing large JPG or H264 encoding this pretty much led to a freeze all the time (one callback per 64K-ish buffer, quickly runs into the race condition). The core of the fix was giving up on that approach and getting the buffer/callback pointers manually after the lock is acquired. I'd also like to take this chance to apologize for being a poor sport and not backporting the changes to picam. My PI has been overseeing my orchard for the last 9 months and I didn't find the time to set up a second unit at home to try out the original picam package + changes. In my case things got more complicated (I'm using it for motion detection, video recording and still captures) hence I switched to JNI for some of the heavier operations and ended up rewriting most of the framework around it. My motion detection needs aside, I've found the biggest drag on performance as oposed to raw raspivid / raspistill comes from the fact JNA allocates a new thread for each callback! You can confirm that easily with a logger or debugger, notice how the thread name always changes. Biggest jump in performance came from adding a Callback manager component on the Java side and the JNI to go with it to supply pointers for operation callbacks. If you ever need to increase throughput for many large images / video, let me know and I'll share some more. Find below a bit of code - the key idea is that the broadcom library usually issues callbacks from the same thread, so when making any calls with callbacks you just supply the pointer to void callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) which then calls back into the java side void receiveCallback(long portAddress, long bufferAddress) BR--Val Java side (simplified):package org.ease.oversight.picamera.mmal.bindings;

import org.ease.oversight.picamera.mmal.bindings.internal.LibInitializer; import com.sun.jna.Pointer;

import java.io.IOException;

/**

• @author vrosca - 26/09/2017 12:41 */ public class LibPiCamera {

  public static void receiveCallback(long portAddress, long bufferAddress) { CallbackManager.receiveCallback(portAddress, bufferAddress); }

  public static native long getCallbackAddress();

  public static native long getReleaseBufferCallbackAddress();

  public static native long jmalloc(int size);

  public static native long jmemcpy(long destination, long source, int size);

  public static native int countDifferences(long dataPointer1, long dataPointer2, int width, int height);

  public static native int countAndMarkDifferences(long target, long source, int width, int height);

  public static native int averageLuma(long image, int width, int height, int stepSize);

  static {

  try {
      LibInitializer.initialize("picamera");
  } catch (IOException e) {
      throw new RuntimeException(e);
  }

  }

  public static void jmemcpy(Pointer destination, Pointer source, int size) {

  jmemcpy(Pointer.nativeValue(destination), Pointer.nativeValue(source), size);

  }

  public static int countDifferences(Pointer dataPointer1, Pointer dataPointer2, int width, int height) {

  return countDifferences(Pointer.nativeValue(dataPointer1), Pointer.nativeValue(dataPointer2), width, height);

  }

  public static int countAndMarkDifferences(Pointer target, Pointer source, int width, int height) {

  return countAndMarkDifferences(Pointer.nativeValue(target), Pointer.nativeValue(source), width, height);

  }

  public static int averageLuma(Pointer image, int width, int height, int stepSize) { return averageLuma(Pointer.nativeValue(image), width, height, stepSize); } }

package org.ease.oversight.picamera.mmal.bindings;

import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_BH_CB_T; import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_T; import com.sun.jna.Pointer; import org.slf4j.Logger; import org.slf4j.LoggerFactory;

import java.util.HashMap; import java.util.Map;

/**

• @author vrosca - 26/09/2017 12:38 */ public class CallbackManager {

  private static final Logger logger = LoggerFactory.getLogger(CallbackManager.class);

  private static final long CALLBACK_ADDRESS = LibPiCamera.getCallbackAddress();

  private static final long RELEASE_BUFFER_CALLBACK_ADDRESS = LibPiCamera.getReleaseBufferCallbackAddress();

  private static final Pointer CALLBACK_POINTER = new Pointer(CALLBACK_ADDRESS);

  private static final Pointer RELEASE_BUFFER_CALLBACK_POINTER = new Pointer(RELEASE_BUFFER_CALLBACK_ADDRESS);

  private static final Map<Long, Callback> callbackMap = new HashMap<>();

  public interface Callback extends MMAL_PORT_BH_CB_T { }

  public static Pointer registerCallback(MMAL_PORT_T port, Callback callback) {

  registerCallback(Pointer.nativeValue(port.getPointer()), callback);
  
  return CALLBACK_POINTER;

  } public static void registerCallback(long address, Callback callback) {

  callbackMap.put(address, callback);

  }

  public static void cancelCallback(MMAL_PORT_T port) {

  cancelCallback(Pointer.nativeValue(port.getPointer()));

  } public static void cancelCallback(long address) {

  callbackMap.remove(address);

  }

  public static void receiveCallback(long portAddress, long bufferAddress) {

  Callback callback = callbackMap.get(portAddress);
  
  if (callback != null) {
  
      callback.apply(new Pointer(portAddress), new Pointer(bufferAddress));
  
  }

  }

  public static Pointer getReleaseBufferCallbackPointer() { return RELEASE_BUFFER_CALLBACK_POINTER; }

}

C side:#include

include

include

include "interface/mmal/mmal.h"

include "interface/mmal/mmal_logging.h"

include "interface/mmal/mmal_buffer.h"

include "interface/mmal/util/mmal_util.h"

include "interface/mmal/util/mmal_util_params.h"

include "interface/mmal/util/mmal_default_components.h"

include "interface/mmal/util/mmal_connection.h"

include "interface/mmal/mmal_parameters_camera.h"

// cached refs for later callbacks JavaVM * g_vm;

jclass callbackDispatcherClass; jmethodID callbackDispatchingMethodId;

jint JNI_OnLoad(JavaVM vm, void reserved) { g_vm = vm; JNIEnv env; if ( (g_vm)->GetEnv(g_vm, &env, JNI_VERSION_1_8) != JNI_OK ) { printf("Can't get JNIEnv, aborting\n"); return -1; } else { jclass clazz = (env)->FindClass(env, "org/ease/oversight/picamera/mmal/bindings/LibPiCamera"); if (clazz == NULL) { printf("Can't find LibPiCamera class, aborting\n"); return -1; } else {      callbackDispatcherClass = (env)->NewGlobalRef(env, clazz); (env)->DeleteLocalRef(env, clazz); callbackDispatchingMethodId = (env)->GetStaticMethodID(env, callbackDispatcherClass, "receiveCallback", "(JJ)V"); if (callbackDispatchingMethodId == NULL) { printf("Can't find receiveCallback static method, aborting\n"); return -1; } else { return JNI_VERSION_1_8; } } } }

pthread_once_t destructorKeyOnce = PTHREAD_ONCE_INIT; pthread_key_t destructorKey;

void callback_destructor(void buf) { unsigned long _tid = buf; printf("Destroying bound thread: %lu\n", _tid); free(buf); (g_vm)->DetachCurrentThread(g_vm); }

void callback_destructor_creator() { pthread_key_create(&destructorKey, callback_destructor); }

void callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) { JNIEnv g_env; int getEnvStat = (g_vm)->GetEnv(g_vm, (void *)&g_env, JNI_VERSION_1_8); if (getEnvStat == JNI_EDETACHED) { // vrosca: associate destructor pthread_once(&destructorKeyOnce, callback_destructor_creator); if ((g_vm)->AttachCurrentThreadAsDaemon(g_vm, (void **) &g_env, NULL) != 0) { printf("JNI Failed to attach\n"); } } else if (getEnvStat == JNI_OK) { // } else if (getEnvStat == JNI_EVERSION) { printf("GetEnv: version not supported\n"); }

(*g_env)->CallStaticVoidMethod(g_env, callbackDispatcherClass, callbackDispatchingMethodId, (jlong)(uintptr_t) port, (jlong)(uintptr_t) buffer);

/ if ((g_env)->ExceptionCheck(g_env)) { (g_env)->ExceptionDescribe(g_env); }/ / if (getEnvStat == JNI_EDETACHED) { (g_vm)->DetachCurrentThread(g_vm); }*/ }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getCallbackAddress(JNIEnv *env, jclass type) { return (jlong)(uintptr_t) &callback; }

static void release_buffer_callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) { mmal_buffer_header_release(buffer); }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getReleaseBufferCallbackAddress(JNIEnv *env, jclass type) { return (jlong)(uintptr_t) &release_buffer_callback; }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmalloc(JNIEnv *env, jclass type, jint size) { return (jlong)(uintptr_t) malloc(size); }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmemcpy(JNIEnv env, jclass type, jlong destination, jlong source, jint size) { //printf("Size: %d", size); return (jlong)(uintptr_t) memcpy((void) destination, (void*) source, size); }

On Wednesday, 21 November 2018, 16:10, Alexander Shniperson <notifications@github.com> wrote:

I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[vrosca]

PS: I use the Oracle JRE on the PI (it has a proper hotspot VM), last year at least OpenJDK was horrendously slow I think they actually ran in interpreted mode on the ARM. With the trick above (and some use of NIO buffers on the Java side) performance absolutely matches raspistill / raspivid

On Wednesday, 21 November 2018, 16:58, Val Rosca <val_rosca@yahoo.com> wrote:

@Mark: glad the changes worked for you! I don't have the thread that fresh in my mind but seem to recall the issue was that JNA was automatically filling the structures (including buffer pointers) BEFORE you acquired the lock from the broadcom libs. When doing large JPG or H264 encoding this pretty much led to a freeze all the time (one callback per 64K-ish buffer, quickly runs into the race condition). The core of the fix was giving up on that approach and getting the buffer/callback pointers manually after the lock is acquired. I'd also like to take this chance to apologize for being a poor sport and not backporting the changes to picam. My PI has been overseeing my orchard for the last 9 months and I didn't find the time to set up a second unit at home to try out the original picam package + changes. In my case things got more complicated (I'm using it for motion detection, video recording and still captures) hence I switched to JNI for some of the heavier operations and ended up rewriting most of the framework around it. My motion detection needs aside, I've found the biggest drag on performance as oposed to raw raspivid / raspistill comes from the fact JNA allocates a new thread for each callback! You can confirm that easily with a logger or debugger, notice how the thread name always changes. Biggest jump in performance came from adding a Callback manager component on the Java side and the JNI to go with it to supply pointers for operation callbacks. If you ever need to increase throughput for many large images / video, let me know and I'll share some more. Find below a bit of code - the key idea is that the broadcom library usually issues callbacks from the same thread, so when making any calls with callbacks you just supply the pointer to void callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) which then calls back into the java side void receiveCallback(long portAddress, long bufferAddress) BR--Val Java side (simplified):package org.ease.oversight.picamera.mmal.bindings;

import org.ease.oversight.picamera.mmal.bindings.internal.LibInitializer; import com.sun.jna.Pointer;

import java.io.IOException;

/**

• @author vrosca - 26/09/2017 12:41 */ public class LibPiCamera {

  public static void receiveCallback(long portAddress, long bufferAddress) { CallbackManager.receiveCallback(portAddress, bufferAddress); }

  public static native long getCallbackAddress();

  public static native long getReleaseBufferCallbackAddress();

  public static native long jmalloc(int size);

  public static native long jmemcpy(long destination, long source, int size);

  public static native int countDifferences(long dataPointer1, long dataPointer2, int width, int height);

  public static native int countAndMarkDifferences(long target, long source, int width, int height);

  public static native int averageLuma(long image, int width, int height, int stepSize);

  static {

  try {
      LibInitializer.initialize("picamera");
  } catch (IOException e) {
      throw new RuntimeException(e);
  }

  }

  public static void jmemcpy(Pointer destination, Pointer source, int size) {

  jmemcpy(Pointer.nativeValue(destination), Pointer.nativeValue(source), size);

  }

  public static int countDifferences(Pointer dataPointer1, Pointer dataPointer2, int width, int height) {

  return countDifferences(Pointer.nativeValue(dataPointer1), Pointer.nativeValue(dataPointer2), width, height);

  }

  public static int countAndMarkDifferences(Pointer target, Pointer source, int width, int height) {

  return countAndMarkDifferences(Pointer.nativeValue(target), Pointer.nativeValue(source), width, height);

  }

  public static int averageLuma(Pointer image, int width, int height, int stepSize) { return averageLuma(Pointer.nativeValue(image), width, height, stepSize); } }

package org.ease.oversight.picamera.mmal.bindings;

import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_BH_CB_T; import org.ease.oversight.picamera.picam.bindings.internal.MMAL_PORT_T; import com.sun.jna.Pointer; import org.slf4j.Logger; import org.slf4j.LoggerFactory;

import java.util.HashMap; import java.util.Map;

/**

• @author vrosca - 26/09/2017 12:38 */ public class CallbackManager {

  private static final Logger logger = LoggerFactory.getLogger(CallbackManager.class);

  private static final long CALLBACK_ADDRESS = LibPiCamera.getCallbackAddress();

  private static final long RELEASE_BUFFER_CALLBACK_ADDRESS = LibPiCamera.getReleaseBufferCallbackAddress();

  private static final Pointer CALLBACK_POINTER = new Pointer(CALLBACK_ADDRESS);

  private static final Pointer RELEASE_BUFFER_CALLBACK_POINTER = new Pointer(RELEASE_BUFFER_CALLBACK_ADDRESS);

  private static final Map<Long, Callback> callbackMap = new HashMap<>();

  public interface Callback extends MMAL_PORT_BH_CB_T { }

  public static Pointer registerCallback(MMAL_PORT_T port, Callback callback) {

  registerCallback(Pointer.nativeValue(port.getPointer()), callback);
  
  return CALLBACK_POINTER;

  } public static void registerCallback(long address, Callback callback) {

  callbackMap.put(address, callback);

  }

  public static void cancelCallback(MMAL_PORT_T port) {

  cancelCallback(Pointer.nativeValue(port.getPointer()));

  } public static void cancelCallback(long address) {

  callbackMap.remove(address);

  }

  public static void receiveCallback(long portAddress, long bufferAddress) {

  Callback callback = callbackMap.get(portAddress);
  
  if (callback != null) {
  
      callback.apply(new Pointer(portAddress), new Pointer(bufferAddress));
  
  }

  }

  public static Pointer getReleaseBufferCallbackPointer() { return RELEASE_BUFFER_CALLBACK_POINTER; }

}

C side:#include

include

include

include "interface/mmal/mmal.h"

include "interface/mmal/mmal_logging.h"

include "interface/mmal/mmal_buffer.h"

include "interface/mmal/util/mmal_util.h"

include "interface/mmal/util/mmal_util_params.h"

include "interface/mmal/util/mmal_default_components.h"

include "interface/mmal/util/mmal_connection.h"

include "interface/mmal/mmal_parameters_camera.h"

// cached refs for later callbacks JavaVM * g_vm;

jclass callbackDispatcherClass; jmethodID callbackDispatchingMethodId;

jint JNI_OnLoad(JavaVM vm, void reserved) { g_vm = vm; JNIEnv env; if ( (g_vm)->GetEnv(g_vm, &env, JNI_VERSION_1_8) != JNI_OK ) { printf("Can't get JNIEnv, aborting\n"); return -1; } else { jclass clazz = (env)->FindClass(env, "org/ease/oversight/picamera/mmal/bindings/LibPiCamera"); if (clazz == NULL) { printf("Can't find LibPiCamera class, aborting\n"); return -1; } else {      callbackDispatcherClass = (env)->NewGlobalRef(env, clazz); (env)->DeleteLocalRef(env, clazz); callbackDispatchingMethodId = (env)->GetStaticMethodID(env, callbackDispatcherClass, "receiveCallback", "(JJ)V"); if (callbackDispatchingMethodId == NULL) { printf("Can't find receiveCallback static method, aborting\n"); return -1; } else { return JNI_VERSION_1_8; } } } }

pthread_once_t destructorKeyOnce = PTHREAD_ONCE_INIT; pthread_key_t destructorKey;

void callback_destructor(void buf) { unsigned long _tid = buf; printf("Destroying bound thread: %lu\n", _tid); free(buf); (g_vm)->DetachCurrentThread(g_vm); }

void callback_destructor_creator() { pthread_key_create(&destructorKey, callback_destructor); }

void callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) { JNIEnv g_env; int getEnvStat = (g_vm)->GetEnv(g_vm, (void *)&g_env, JNI_VERSION_1_8); if (getEnvStat == JNI_EDETACHED) { // vrosca: associate destructor pthread_once(&destructorKeyOnce, callback_destructor_creator); if ((g_vm)->AttachCurrentThreadAsDaemon(g_vm, (void **) &g_env, NULL) != 0) { printf("JNI Failed to attach\n"); } } else if (getEnvStat == JNI_OK) { // } else if (getEnvStat == JNI_EVERSION) { printf("GetEnv: version not supported\n"); }

(*g_env)->CallStaticVoidMethod(g_env, callbackDispatcherClass, callbackDispatchingMethodId, (jlong)(uintptr_t) port, (jlong)(uintptr_t) buffer);

/ if ((g_env)->ExceptionCheck(g_env)) { (g_env)->ExceptionDescribe(g_env); }/ / if (getEnvStat == JNI_EDETACHED) { (g_vm)->DetachCurrentThread(g_vm); }*/ }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getCallbackAddress(JNIEnv *env, jclass type) { return (jlong)(uintptr_t) &callback; }

static void release_buffer_callback(MMAL_PORT_T port, MMAL_BUFFER_HEADER_T buffer) { mmal_buffer_header_release(buffer); }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_getReleaseBufferCallbackAddress(JNIEnv *env, jclass type) { return (jlong)(uintptr_t) &release_buffer_callback; }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmalloc(JNIEnv *env, jclass type, jint size) { return (jlong)(uintptr_t) malloc(size); }

JNIEXPORT jlong JNICALL Java_org_ease_oversight_picamera_mmal_bindings_LibPiCamera_jmemcpy(JNIEnv env, jclass type, jlong destination, jlong source, jint size) { //printf("Size: %d", size); return (jlong)(uintptr_t) memcpy((void) destination, (void*) source, size); }

On Wednesday, 21 November 2018, 16:10, Alexander Shniperson <notifications@github.com> wrote:

I agree, native applications will always be faster, but in our hands to bring the speed close to the native, we (community) just need to try— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[caprica]

My understanding is that JNA does indeed allocate a new Java Thread object for each callback, but no actual thread gets allocated, so that's not as bad as it might initially seem.

And yes, your comment about getting the structures filled before them being locked now makes sense. Thanks a lot.

I do want to reiterate that initially the main goal of this project was to make it very easy for inexperienced developers to use the Pi camera (Pi is foremost a learning platform after all), and for me that meant just using Java without asking people to compile JNI wrappers and such.

If this issue is truly vanquished, I'm more motivated to advance this project now so if there are things that we can do to improve performance (or other things) I would like to hear about it.

[caprica]

In hindsight, as is often the case, this bug is obvious and stupid.

In short, by using the JNA auto-mapped object in the callback method signature, the native memory was being read by the Java code before that memory buffer was explicitly locked. Clearly this is bad, m'kay.

Anyway, I have now captured literally thousands of pictures of varying sizes from small up to 10mb without a single problem during my most recent testing and I will commit the fix today.

[vrosca]

In my case it hasn't locked once in 9 months of operation (pi zero, 1 cpu) - hopefully we're not overlooking some other configuration where it might fail. A JNI pthread attachment and custom callback logic is overkill for most applications (had to squeeze every bit of performance I could to run motion detection on the pi zero at 8 Mp). That said, once you profile the callbacks you'll probably notice the same thing I did: with the custom callback and using direct bytebuffers performance and cpu usage is identical to raspicam / raspivid. In my case, I bundle the so in the jar and have a simple class that extracts it in a temp file and loads the library from there. Since there aren't that many pi configurations out there, cross-compiling it in a makefile isn't that hard but yeah it does add to overall complexity... Happy holidays ahead !

On Wednesday, 21 November 2018, 18:48, Mark Lee <notifications@github.com> wrote:

In hindsight, as is often the case, this bug is obvious and stupid.In short, by using the JNA auto-mapped object in the callback method signature, the native memory was being read by the Java code before that memory buffer was explicitly locked. Clearly this is bad, m'kay.Anyway, I have now captured literally thousands of pictures of varying sizes from small up to 10mb without a single problem during my most recent testing and I will commit the fix today.— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[caprica]

Fixed at 32deac5974fa87cecf8c064d0fdefe53f2583467.