Various errors with the high level Java API

[mclem]

Hello,

I am trying to the high level Java API but I cannot make it work consistently (my program is often stuck in the TwainSource.acquire()method). The target device is an Epson Perfection V600 but in my local development environment I am using the TWAIN2 sample DS. The DTWDEMO.exe always works flawlessly so I suspect the problems are in the Java API or the JNI layer.

Here are some of the problems I encountered:

1. Acquiring a PDF file

With DTWDEMO: twain_pdf_demo.log Running PDFDemo.java: twain_pdf_jni.log

When I click on "Scan" the UI becomes stuck. I cannot close it by clicking on "Cancel".

There are a lot of errors and exceptions in the JNI log. I also found these strange values in the log:

TW_MEMREF is TW_IMAGELAYOUT:
{
    Frame=0,0-0,0
    DocmentNumber=4294967295
    PageNumber=4294967295
    FrameNumber=4294967295
}

[2024-08-30 11:45:06] Thread [23412] Output: return code from DSM=1 (TWRC_FAILURE)

I noticed that the TW_FRAME.isFrameValid() method inacurrately returns true when all values are set to 0, because of the comparison with Double.MIN_VALUE: If I define a frame size (ts.getAcquireCharacteristics().getPagesSupportOptions().getFrame().setRight(6).setBottom(10);) then it looks like the acquisition goes further but it does not complete too: twain_pdf_jni+frame.log

=> It seems impossible to acquire a file (even a simple BMP with TinyApplication.java)

2. Acquiring to buffer

I am able to run the BufferAcquisitionDemo.java demo, however if I add a logger and enable it with twainSession.enableLogging(true) then the program stays stuck in the acquire method: twain_buf_jni+logging.log Now if I disable logging just before calling acquire the demo works correctly: twain_buf_jni+notlogging.log

=> It is possible to acquire to buffer, but not if I collect logs

3. With the EPSON V600

I have not been able to make a lot of tests yet, but I noticed that:

• When I try to add a logger, the source does not open (somestimes error -1039, sometimes I see this error log: https://github.com/dynarithmic/twain_library_source/blob/53a5e00d087464c3be7b6cf26f6a705b7adfbe6c/source/h/arrayfactory.h#L147)
• A black line appears at the top of the scanned images using AcquireType.BUFFER (there is no such thing when I use the EPSON Scan driver directly). Maybe this is linked to the fact that I added ac.getPagesSupportOptions().getFrame().setLeft(0).setTop(0).setRight(8.3).setBottom(11.7); to scan an A4 paper (not setting the frame resulted in an error).

[dynarithmic]

Thanks for the detailed message.

It's strange why the sample source would get stuck like this. I have not had an issue with the sample TWAIN data source.

As to the logs and the amount of information generated, there are a lot of calls to DTWAIN_IsTwainMsg which can be removed by specifying the appropriate logging flags. You can adjust the java source code here and remove the SHOW_ISTWAINMSG flag in the logging options (probably by doing a LOGALL & ~SHOW_ISTWAINMSG, if I have my boolean logic right).

The log files should be much smaller after doing this, but still retain the call stack and other TWAIN traffic.

Given this, try the following suggestions:

1) Use the latest version 5.4.8 of the DTWAIN libraries (you will also need to use the text resources, since there have been changes).

2) You can also try the library in the development branch to see if any of the updates there makes a difference.

3) You can also try the unreleased version 5.5 of the DTWAIN DLL's, and the updated text resources.

I believe the exception error you received was addressed in the experimental 5.5 version, or if not may have been addressed in 5.4.8.

[mclem]

Hi, and thanks for the reply,

I made the following modifications:

• Download DTWAIN 5.5.0 from twain_library:master-stating
• Download the latest JNI dll from twain_library-java:development
• Change MAXIMUM verbosity level to LoggingOptions.LOGALL.value() & ~LoggingOptions.DECODE_SOURCE.value() & ~LoggingOptions.DECODE_DEST.value() & ~LoggingOptions.SHOW_ISTWAINMSG.value()

Following are my observations, considering:

• "logging" means starting the session like this

  TwainSession twainSession = new TwainSession(SessionStartupMode.NONE);
  twainSession.enableLogging(true);
  TwainSession.getLogger()
      .setVerbosity(LoggerVerbosity.MAXIMUM)
      .addLogger(new TwainFileLogger(Paths.get(System.getProperty("user.dir")).resolve("twain_jni.log").toAbsolutePath().toString()));
  twainSession.start();

• "logging stopped" means that I setup logging then call twainSession.stopLogging() right before twainSource.acquire()
• "callback" means that I register a TwainCallback that simply prints the name of the called method (e.g. System.out.printl("onAcquireStarted"))

A) Acquiring to file with TinyApplication: A1) Run, wait for the TWAIN UI, click "Cancel" => the demo completes normally with no acquisition, DTWAIN_AcquireFile returns 1002 A2) Run, wait for the TWAIN UI, click "Scan" => the UI is stuck (cannot close or cancel), DTWAIN_AcquireFile never returns A3) Retry A2) with logging => the UI is stuck (cannot close or cancel), DTWAIN_AcquireFile never returns => twain_jni_A3.log A4) Retry A2) with callback => after the UI opens I see: onGeneralError, onGeneralError

B) Acquiring to memory with BufferedAcquisitionDemo: B1) Run, wait for the TWAIN UI, click "Cancel => the demo completes normally with no acquisition, DTWAIN_AcquireBuffered returns no image bytes B2) Run, wait for the TWAIN UI, click "Scan" then "Cancel => the demo completes normally with one acquisition, DTWAIN_AcquireBuffered returns the image bytes B3) Retry B2) with logging => the UI is stuck (cannot close or cancel), DTWAIN_AcquireBuffered never returns => twain_jni_B3.log B4) Retry B2) with logging stopped => the demo completes normally with one acquisition, DTWAIN_AcquireBuffered returns the image bytes => twain_jni_B4.log B5) Retry B2) with callback => after the UI opens I see: onGeneralError, onGeneralError, onGeneralError, onProcessDIBAccepted, onUIOpened, onAcquireStarted B6) Retry B2) with logging and callback => after the UI opens I see: onGeneralError B5) Retry B2) with logging stopped and callback => after the UI opens I see: onGeneralError, onGeneralError, onGeneralError, onProcessDIBAccepted, onUIOpened, onAcquireStarted

Concerning log file B3:

• I selected the source around 10:29:45
• I clicked on scan around 10:29:47
• I clicked on cancel around 10:29:51

In summary:

• I cannot capture to a file even with the simplest example, and the logs show some errors related to the file system. However capturing to a file works with DTWDEMO.exe
• It looks like enabling logging prevents something from initializing correctly, as when logging is disabled some callbacks are called after the UI is shown: onProcessDIBAccepted, onUIOpened and onAcquireStarted.
• In the JNI logs there are a lot of "DTWAIN Exception Error" and "Callback did not work" occurrences

[dynarithmic]

1) Please try the latest DLL's in the experimental branch. From looking at the logs, you are probably not running the latest DLL's that contain various fixes to the opening of a Source, plus the file system changes. The "DTWAIN Exception Error" messages also should be minimized, if not completely gone.

2) As to the "file system", it is actually the internal TWAIN file system that some sources support (usually high-end duplex scanners). I tested with a sample Kodak virtual scanner that emulates the file system, and you should not be receiving the exception error messages if your scanner doesn't support it.

3) I tried the "SimpleLoggingDemo.java", and found no issues with logging to the Java console. The entire log when I run the program is here: SimpleLoggingDemo.log

4) I suggest turning off the LoggingOptions.DECODE_TWEVENT to further reduce the size of the log. You would do this similar to how you turned the SHOW_ISTWAINMSG logging off.

5) If possible, can you attach your entire Java program that you are running? This is to ensure that I have the exact program that is giving the issue.

6) The logs just seem to stop right in the middle of an operation. What happens if you only log to the console and not a file?

[mclem]

Hi,

1. OK, I initially downloaded the 5.5.0 version from twain_library instead of twain_library-source. The logs now show "Internal Build Number: 000"

2. There are now less exceptions about the file system. For example in the buffered acquisition scenario I do not see "Entering FSGetCamera" anymore.

3. I also tried to run the SimpleLoggingDemo without changing anything. Comparing your log with mine I found these differences:

   • You have the Sample DS version 2.4.0 whereas I have version 2.5.0
   • In my log, there is no trace of CAP_SUPPORTEDDATS

   • In my log, each callback leads to an exception, e.g.:

     [2024-09-03 10:10:43.426] Thread [34372] To callback:
     DTWAIN Message(HWND = 00000000, MSG = 50372, Notification code = 1038, LPARAM = -1058
     
     [2024-09-03 10:10:43.427] Thread [34372] **** DTWAIN Exception Error ****.  Module: DTWAIN_EnumCamerasEx

   • My log nevers resumes as the program is stuck in the acquire method

4. Here are 4 less verbose logs where I also compacted conse

   • Acquire buffered: twain_jni_buffered.log cutives ocurrences of the TWRC_NOTDSEVENT messages:
   • Acquire buffered with logging stopped before acquisition twain_jni_buffered+stopped.log
   • Acquire file: twain_jni_file.log
   • Acquire file with logging stopped before acquisition: twain_jni_file+stopped.log

   There are a lot of errors about callbacks in the files where the acquisition is logged

5. Here you will find the programs that I used to generate these logs: https://github.com/mclem/twain_library-java/commit/ebf7520da79ddfee3d8e7e0a56861825e31445e4

6. If I log to the console I have the complete log messages as expected. It looks like a buffer is not flushed when stopping logs to file, so some lines are lost. I completed my log files with the additional lines from the console.

[dynarithmic]

1) There are newer versions of the JNI DLL's in the development repo. I believe they should address the DTWAIN Exception messages you are seeing now.

2) In newlog.log, there are 4 places where CAP_SUPPORTEDDATS is being processed:

  Line 2145     CAP_SUPPORTEDDATS
  Line 2330     Array[10] = CAP_SUPPORTEDDATS
  Line 3379     Cap=CAP_SUPPORTEDDATS
  Line 3420     Cap=CAP_SUPPORTEDDATS

3) Since it seems that console logging works and file logging does not work, this seems to be a Java (and not a JNI) issue. The JNI code does not know that Java is writing to the console or a file. The JNI layer calls the Java method TwainLogger.logMessage with the log message, and from there Java takes over.

Note that TwainLogger keeps an array of TwainLoggerProc instances, and both TwainConsoleLogger and TwainFileLogger are derived from TwainLoggerProc. So there is something going on, at least in your environment, that is causing TwainFileLogger to cause an issue.

I think you should debug what is going on in TwainFileLogger, since I cannot duplicate the error. Quite possibly, you can derive your own logger from TwainLoggerProc and perform the file writing in a different way than what is presented in TwainFileLogger.java .

[mclem]

1. I am indeed using the dtwainjni32u DLL from https://github.com/dynarithmic/twain_library-java/blob/development/JNI_DLL/32bit/dll32.zip => version 1.0.4.0, modified 01/09/2024 16:55.

2. We can forget about CAP_SUPPORTEDDATS, I missed it when I tried to compare both log files line by line. The only meaningful difference seems to be that callbacks never work in my environment according to my logs.

3. About the TwainFileLogger missing contents, the problem is simply that it uses a FileWriter which is buffered. Thus when the Log method receives a messages and calls logFile.write(sMsg + "\n");, the line is not directly appended to the file. Just for testing, I added a call to FileWriter.flush after writing each message and now the log file is complete: https://github.com/dynarithmic/twain_library-java/commit/f28a4a4dd34adcbbaa733d19089c5685c6aedd30

Some more observations:

• There are no errors/exceptions on the Java side, the Sample DS UI is simply stuck once I click on "Scan".
• If I simply comment out session.enableLogging(true); in my programs (in SessionFactory) then for example AcquireBufferedWithLog does not become stuck anymore when I click on "Scan"
• This made me investigate the TwainLogger class: if I comment out the USE_CALLBACK flag here then the UI is not stuck anymore after I click on "Scan", but I have no more logs to analyze
• I then completely removed the TwainLogger setup and replaced it by adding calls to session.getAPIHandle().DTWAIN_SetTwainLog(0x0000FFFF, "native.log"); and session.getAPIHandle().DTWAIN_EnableMsgNotify(1); right after starting the TWAIN session: the UI does not hang anymore
• I noticed that my printing callback also does not print "onProcessDIBAccepted", "onUIOpened" and "onAcquireStarted" when the logger is setup with TwainLogger, so problems must occur even before I click on the "Scan" button.

In summary, in my environement (Eclipse Adoptium JDK 17.0.11.9) :

• The buffered acquisition works only if I do not add Java logs or if I add "native" logs

  • This is the log file when setup from the Java API: twain_jni_buffered.log
  • This is the log file when setup with SetTwainFile: twain_native_buffered.log

• The file acquisition never works, however I was able to capture "native" logs which also contains "Callback did not work" errors: twain_native_file.log

  By adding source.getAcquireCharacteristics().getPagesSupportOptions().getFrame().setLeft(0).setTop(0).setRight(8.3).setBottom(11.7); I was able to make one "DTWAIN Exception Error" disappear when setting ICAP_FRAMES (else the frame has size {left=0},{top=0},{right=0},{bottom=0} which is reported as invalid)

[dynarithmic]

Thanks for the detailed response.

I will look at the changes you made on the Java side with the TwainFileLogger.

The logger in general is really run purely by callback method on the JNI side (note the calls to DTWAIN_SetCallback in the JNI code). There is actually no direct call to DTWAIN_SetTwainLog to set up the logging. The idea is that the JNI code should only send the message string back to Java, and let Java handle what it should do with the string.

Of course, you can go directly to the native method of setting the twain log, and that should work if you want to log to a file. However you should then refer to the general DTWAIN documentation on DTWAIN_SetTwainLog, as the flags may differ slightly from what you see in Java.

Also, I am looking at the UIOPENING notification, and there definitely looks like there is an issue that I need to fix in the DTWAIN library to clear up the confusion between that notification and other notifications. The Java code is simply showing the symptoms of this.

Since version 5.5 of DTWAIN is still in development phase, I will try and get updates to the library, and then you can get the latest from the development repo.

[dynarithmic]

• I added the latest DTWAIN 5.5.0 (Build 001) dlls here. These should alleviate some of the General Error notifications you were receiving, plus straighten out the UIOPENING notification.
• I updated the Java source files and the dtwain.jar file to add the UIOPENING plus the new Big Tiff file support. You can get those changes here.
• The ProcessDIBAccepted notification is only sent if you return a 0 from the OnProcessDIB notification (meaning that you do not want DTWAIN to further process the image that was returned by the device). By default, DTWAIN will attempt to resample the image, rescale, or negate the image if necessary.

I didn't add the TwainFileLogger changes you mentioned just yet -- I want to make sure that the notifications you were not getting are now being received.

[mclem]

Hi

The logger in general is really run purely by callback method on the JNI side (note the calls to DTWAIN_SetCallback in the JNI code). There is actually no direct call to DTWAIN_SetTwainLog to set up the logging. The idea is that the JNI code should only send the message string back to Java, and let Java handle what it should do with the string.

OK, I had the idea to play with DTWAIN_SetTwainLog because I saw this line in TwainLogger.java

I now have better results with the experimental 5.5.0 build 001 so I have updated and simplified my test programs:: https://github.com/dynarithmic/twain_library-java/commit/703f72494c1487f202aea73a7690b1fdc3c2308d In summary:

• I can now collect logs in Java without blocking the acquisition
• I can now acquire to a file (tried the default BMP and PDF)

Here are the logs:

• Acquire buffered with default settings: dtwain_AcquireBuffered.log
• Acquire buffered no UI: dtwain_AcquireBufferedNoUi.log
• Acquire file with default settings: dtwain_AcquireFile.log
• Acquire file no UI: dtwain_AcquireFileNoUi.log
• Acquire file PDF: dtwain_AcquireFilePdf.log

However there seem to be some remaining problems:

• The Java TwainCallback methods are not called if a Java logger is enabled during the acquisition. Using the Java debugger I see that onTwainEvent is called once by DTWAIN_SysInitialize and then the breakpoint is never hit anymore. I see nothing new in the logs apart from a lot of "Callback did not work..." messages. Here are the native logs when there is no Java logger (dtwain_AcquireBufferedNoUi_nativelogging.log) and here are the Java logs when I call stopLogging right before the acquisition (dtwain_AcquireBufferedNoUi_loggingdisabled.log). It seems linked to the JNI calls to DTWAIN_SetTwainLog(verbosity, "").
• A minor one: by default, ImageTypeOptions.bitDepth is set to 0. When preparing the acquisition, the OptionsApplier class tries to set this capability without checking if it has been changed like it is done for other capabilities: this produces a TWCC_BADVALUE error in the logs

Thanks for all the updates :)

[dynarithmic]

I looked at your logs, and see the "Callback did not work..." message. That message is generated within the DTWAIN code here.

I took your code and ran it against the TWAIN source 2.4. I do not get these messages, indicating that possibly something is happening between JNI and Java that is causing an exception to be thrown in your environment. Not sure, since it looks like all of the logging messages for TN_UIOPENING are being written to your log file, but some quirk is throwing an exception on your end.

The exception doesn't seem to be "harmful", just annoying.

Here is my log file: my_AcquireBuffered.log

I am running JRE 1.8.0_34.

To this, I will make changes to only log the exception error if the MISCELLANEOUS flag is on for logging. Right now, you will always receive this message if an exception is thrown.

Thanks for the information on the bit depth setting. I will also make changes there to test for the default bit depth of 0.

[mclem]

I think there is some kind of conflict between the "basic" JNI callback (DTWAINCallback / TwainCallback.onTwainEvent) and the "logger" JNI callback (DTWAINLoggerCallback / TwainSession.logEvent). In Java 8 everything works correctly but starting from Java 9 the basic callback does not work anymore if Java logs are enabled.

I recompiled the JNI dll locally with some printing statements to try to find an explanation. Here is a gdb trace that I collected when the callback fails if it helps:

Thread 4 received signal SIGSEGV, Segmentation fault.
0x0000000074e2cb3c in jvm!AsyncGetCallTrace () from C:\DEV\JDK\jdk-9.0.4\bin\server\jvm.dll
(gdb) bt
#0  0x0000000074e2cb3c in jvm!AsyncGetCallTrace () from C:\DEV\JDK\jdk-9.0.4\bin\server\jvm.dll
No symbol table info available.
#1  0x00007ffaba60cb5a in JNIEnv_::CallStaticIntMethod (this=0x2d135ccf1d8, clazz=0x2d1663cb358, methodID=0x2d166d59338) at C:/Program Files/Eclipse Adoptium/jdk-17.0.9.9-hotspot/include/jni.h:1451
        args = 0xe8400fb418 "╗\v"
        result = 0
#2  0x00007ffaba571a2c in JavaCallback::DTWAINCallback (w=3003, l=3098388890768, This=1732340928) at ../DTWAINJavaAPI.cpp:217
        sGeneralErrors = std::set with 1 element = {[0] = 1038}
        pCallback = std::shared_ptr<JavaCallback> (use count 3, weak count 0) = {get() = 0x2d167416cc0}
        iter = {first = "TwainMessageListener", second = {m_jClassName = "com/dynarithmic/twain/highlevel/TwainCallback", m_jFunctionName = "(IJZ)I", m_jFunctionSig = "", m_jCallbackClass = 0x2d1663cb358, m_jCallbackMethodID = 0x2d166d59338}}
        pCallInfo = @0x2d167406180: {m_jClassName = "com/dynarithmic/twain/highlevel/TwainCallback", m_jFunctionName = "(IJZ)I", m_jFunctionSig = "", m_jCallbackClass = 0x2d1663cb358, m_jCallbackMethodID = 0x2d166d59338}
        wrapper = {m_pCallBack = std::shared_ptr<JavaCallback> (use count 3, weak count 0) = {get() = 0x2d167416cc0}}
        pEnv = 0x2d135ccf1d8
        retval = 0
#3  0x00007ffab9e66e04 in DTWAIN_CheckHandles () from C:\DEV\JNI\dtwain64u.dll
No symbol table info available.
#4  0x00007ffab9e6821b in DTWAIN_RemoveCallback () from C:\DEV\JNI\dtwain64u.dll
No symbol table info available.
#5  0x00007ffab9e6bfbd in DTWAIN_WindowProc () from C:\DEV\JNI\dtwain64u.dll
No symbol table info available.
#6  0x00007ffb3cf4ef5c in USER32!CallWindowProcW () from C:\WINDOWS\System32\user32.dll
No symbol table info available.
#7  0x00007ffb3cf4dfbb in USER32!SendMessageW () from C:\WINDOWS\System32\user32.dll
No symbol table info available.
#8  0x00007ffb3cf4d814 in USER32!SendMessageW () from C:\WINDOWS\System32\user32.dll
No symbol table info available.
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

At first I thought it may have something to do with this line, as if I understood correctly the code runs like this:

• DTWAINCallback: JNIEnv* pEnv = pCallback->getJNIEnv(); => attach the thread to the JVM
• DTWAINCallback: call DTWAIN_IsSourceValid which triggers a log:
  • DTWAINLoggerCallback: JNIEnv* pEnv = pCallback->getJNIEnv(); => attach the thread to the JVM (no-op)
  • DTWAINLoggerCallback: call the static Java method
  • DTWAINCallback: destroy JavaCallbackWrapper wrapper(pCallback); => detach the thread
• DTWAINCallback: call the static Java method => is the JNIEnv pointer still valid given we called detach?
• DTWAINCallback: destroy JavaCallbackWrapper wrapper(pCallback); => detach the thread

However commenting out the call to IsSourceValid (I hardcoded FALSE) did not change anything so I must be wrong...

Then I played with the DTWAINLoggerCallback and I found out that if I replace CreateJStringFromCString(pEnv, str) with NULL, the basic callback works again. Maybe some kind of memory leak as some sources suggest that these strings should be freed at some point:

• https://groups.google.com/g/android-ndk/c/6QyOGxVTSWU
• https://forums.oracle.com/ords/apexds/post/newstringutf-and-references-from-calls-to-java-5809
  
  In summary, using the original JNI dll from the development branch I found that:
• At log verbosity LOGALL (0xFFFF) the basic callback does not work (it works only once at startup, when w=1150)
• If I disable SHOW_CALLSTACK, SHOW_LOWLEVELTWAIN, SHOW_NOTIFICATIONS and SHOW_MISCELLANEOUS then the basic callback works again (but then almost no logs reach the Java application, so it is similar as stopping the logs).
• When not using the Java logs but the native logs (by calling DTWAIN_SetTwainLog(verbosity, filePath)), the basic callback works even if the verbosity is LOGALL

[dynarithmic]

Thanks for the information.

I did some debugging, and I ran using Java 11. I duplicated the issue that you were seeing with the exceptions being thrown instead of having the callback being called. The callbacks do work right before the Source is selected, but from there, they stop working and instead exceptions are thrown when opening the UI, etc.

For a test, I changed the JNI code here in the constructor and created another member function called InitInfo:

JavaCallback::JavaCallback(JNIEnv * pEnv) : JavaAdapter(pEnv)
{
    InitInfo(pEnv);
}

void JavaCallback::InitInfo(JNIEnv* pEnv)
{
    Init(pEnv);
    // Logger and listener callbacks are initialized here
    std::array<std::string, 2> categories = { "TwainLogger", "TwainMessageListener" };
    for (auto& s : categories)
    {
        auto iter = JavaFunctionNameMapInstance::getFunctionMap().find(s);
        auto& callback_info = iter->second;
        auto& callback_fn = iter->second.m_mapFunctions;
        auto& namesig = (*callback_fn.begin()).second;
        auto& jcallback = m_jCallbackInfo.insert({ s, {callback_info.m_className, namesig.funcName, namesig.funcSig} }).first;

        // Get the java class class
        jcallback->second.m_jCallbackClass = pEnv->FindClass(jcallback->second.m_jClassName.c_str());

        // get the method id stored in the class
        jcallback->second.m_jCallbackMethodID = pEnv->GetStaticMethodID(jcallback->second.m_jCallbackClass,
            namesig.funcName.c_str(),
            namesig.funcSig.c_str());
    }
    m_pThisObject = this;
}

Now InitInfo is called right before each call to the callback here.

    pCallback->InitInfo(pEnv);
    if ( !sGeneralErrors.count(w))
    {
        retval = pEnv->CallStaticIntMethod(pCallInfo.m_jCallbackClass, pCallInfo.m_jCallbackMethodID, w, static_cast<jlong>(l),
                                           API_INSTANCE DTWAIN_IsSourceValid(reinterpret_cast<DTWAIN_SOURCE>(l))?TRUE:FALSE);
    }
    else
        retval = pEnv->CallStaticIntMethod(pCallInfo.m_jCallbackClass, pCallInfo.m_jCallbackMethodID, w, 0, FALSE);

From there, the callbacks are now invoked, and the exceptions are now gone. The code always gets the latest pointer to the JVM environment on each call, so that wasn't an issue. As a matter of fact, the environment pointer is stable. The issue is that the class and method id's can change on each call. This is highly unfortunate.

Since a lot of the code right now relies on method ID's to call back to the Java's com.dynarithmic.twain.lowlevel functions, this would mean having to reinitialize the function pointers whenever a call from the JNI side needs to be made. This would be highly inconvenient in terms of efficiency (the function in the JNI code to read and process dtwainjava.info is where the function and class id's are established) just to ensure the class and method ID's are changed.

Admittedly, I don't know if there is a JVM setting that state "don't move my classes once they are loaded" -- maybe something to do with the classloader and/or for some reason, the callback objects were garbage collected? If you know if there is a way to do this, then that would probably solve the problem, at least temporarily until I find a way to better ensure that the class and method id's are loaded whenever a call is made.

[dynarithmic]

After going through the code again, I think it will be ok. There is this function that is called every time a JNI object that mimics the Java low-level object is called.

Basically that function takes the class name, finds the function name and signature in the map that was previously populated, and re-establishes all of the method id and class id info. The issue seems to be that the static methods (logger and callback) were never re-initialized.

I will try and get new JNI DLL's in development.

[dynarithmic]

New commits done here. I tested this with Java 11 and using the AcquireBuffered() program. I didn't come across the issue with the exceptions being thrown.

[mclem]

Admittedly, I don't know if there is a JVM setting that state "don't move my classes once they are loaded" -- maybe something to do with the classloader and/or for some reason, the callback objects were garbage collected? If you know if there is a way to do this, then that would probably solve the problem, at least temporarily until I find a way to better ensure that the class and method id's are loaded whenever a call is made.

According to the following links I understood that if you want to reuse safely a java methodID, you should keep a global reference to its class to avoid it being unloaded:

• https://docs.oracle.com/en/java/javase/11/docs/specs/jni/design.html#accessing-fields-and-methods
• https://stackoverflow.com/questions/2093112/why-i-should-not-reuse-a-jclass-and-or-jmethodid-in-jni
• https://stackoverflow.com/questions/10617735/in-jni-how-do-i-cache-the-class-methodid-and-fieldids-per-ibms-performance-r/13940735#13940735

To test, I replaced this line with the following code and everything seems to work correctly:

jclass tempLocalClassRef = pEnv->FindClass(jcallback->second.m_jClassName.c_str());
jclass globalClassRef = (jclass) pEnv->NewGlobalRef(tempLocalClassRef);
pEnv->DeleteLocalRef(tempLocalClassRef);
jcallback->second.m_jCallbackClass = globalClassRef;

 
 
Also while compiling the JNI layer I noticed a mistake in the javaobjetcaller.h file: https://github.com/dynarithmic/twain_library-java/blob/81ab5544edec62cd06e224caf9bafae3c6304e98/JNI_Source/classes/com/dynarithmic/jnicode/javaobjectcaller.h#L734 It should be getFunctionName instead of GetProductName.

[dynarithmic]

Thanks. I made the correction to the getFunctionName.

From the first document link, I see this:

A field or method ID does not prevent the VM from unloading the class from which the ID has been derived. After the class is unloaded, the method or field ID becomes invalid. The native code, therefore, must make sure to:

keep a live reference to the underlying class, or
recompute the method or field ID

So the second option of recomputing the method ID's is what keeps the other functions from not failing in the current JNI code (except for the logger and callback methods).

I took the code you have for creating the global ref for the logger and callback, and things now work correctly. I will probably use this code for the static methods, and keep the reinitialization code for the other classes.

[dynarithmic]

New commit to master (#17). This should address most of the issues you've seen.

[mclem]

Hi, I can confirm that everything works correctly with the latest JNI and library DLLs. Thank you so much for the help and quick feedback! 😃