Setting Brightness, Contrast, Rotation, Auto Deskew, etc not working with pixel type TWAIN_PT_BW (black & white)

[sean-neeley]

I'm using a .NET app with DTWAIN 5.2.0.8. The application has an option to scan color or black & white. For black & white, this call is made:

TwainAPI.DTWAIN_SetPixelType(selectedSource, TwainAPI.DTWAIN_PT_BW, 1, true);

However, when scanning black and white, it seems that these DTWAIN calls do nothing:

TwainAPI.DTWAIN_SetBrightness() TwainAPI.DTWAIN_SetContrast() TwainAPI.DTWAIN_SetRotation() TwainAPI.DTWAIN_EnableAutoDeskew()

There may be others that don't work (border detect and blank page detect?). I checked the return values of these calls and they are returning success. When I switch to color with:

TwainAPI.DTWAIN_SetPixelType(selectedSource, TwainAPI.DTWAIN_PT_RGB, 24, true);

They do what they are expected to do. Are there any limitations with black & white scanning?

[dynarithmic]

Please note that all of those calls depend on the device and how the TWAIN driver is implemented.

The DTWAIN library makes calls to TWAIN DSM, which then sends that information to the device. If the device returns success, but does nothing, that is a bug in the driver for the device.

To diagnose this issue better, you will need to

1) Identify the device that is not working properly 2) Generate a log (DTWAIN_SetTwainLog) that records all calls made between DTWAIN, TWAIN, and the driver.

Right now, it is impossible to diagnose these issues without this information.

[dynarithmic]

Also, a small minimal example should be posted, to ensure you are making the calls in the correct manner.

In addition, if you are attempting to scan without using a user-interface, then all bets are off if you don't know what the capabilities are to set to get the B/W working correctly, or if these options can be turned on or off without a user-interface. Is the auto-deskew, etc. available for B/W when showing the user interface?

This goes back to the earlier comment -- the driver is responsible for all the inner workings of the device, and hopefully the manufacturer has implemented all or most of their features in a way where an application can access them via TWAIN.

I have seen drivers not get the non-UI version of their device working 100% the same as when a UI is shown, so this isn't strange thing -- it happens. The most you can do in this case is to create the simplest program possible, and see if you can get any further with it (in addition to creating logs, as I mentioned in the previous comment).

[sean-neeley]

1. Identify the device that is not working properly
2. Generate a log (DTWAIN_SetTwainLog) that records all calls made between DTWAIN, TWAIN, and the driver.

The device is an HP Scanjet 7000 S3. I generated a log from our program with a black and white scan that should rotate the image 180 degrees. The rotation did not occur as expected. The attached log is TwainLog-BlackWhite.txt. I then tried the same thing in our application in color mode and attached the log. I'm not familiar with the DTWAIN log output, so hopefully you would be able to look it over. I appreciate the help.

TwainLog-Color.txt TwainLog-BlackWhite.txt

[sean-neeley]

This goes back to the earlier comment -- the driver is responsible for all the inner workings of the device, and hopefully the manufacturer has implemented all or most of their features in a way where an application can access them via TWAIN.

I have seen drivers not get the non-UI version of their device working 100% the same as when a UI is shown, so this isn't strange thing -- it happens. The most you can do in this case is to create the simplest program possible, and see if you can get any further with it (in addition to creating logs, as I mentioned in the previous comment).

I was able to enable the UI and show it. I found that the scanner was set to use a "Faster Black & White" profile. There is also a regular "Black & White" profile. The Faster Black & White profile has many features disabled in the UI, including rotation.

So that looks to be the source of our issue. We need to make sure the HP is not set to the Faster mode. Hopefully that is something I can figure out how to accomplish programatically.

[dynarithmic]

OK, that's good.

I have an HP scanner with a similar (probably same) interface. From the capabilities that are available, I don't see one that would select a profile.

Basically I am looking for:

1) custom capabilities (capabilities described as CAP_CUSTOMBASE + yyyy, where yyyy is the cap custom number. or 2) The capability CAP_CUSTOMDSDATA, which allows the device to describe settings that you can set/get.

I was hoping for 2) to be available, unfortunately the scanner driver I am using doesn't support CAP_CUSTOMDSDATA. If it did support it, the DTWAIN_GetCustomDSData and DTWAIN_SetCiustomDSData functions could have been used to see if it affected the profile setting.

For 1) There are no custom capabilities available, If there were custom caps available, it would be a matter of experimenting setting them to see if they have any effect.

For both 1) and 2), it is manufacturer dependent on what the custom data and custom caps will do, so I can't tell you what will happen if they existed for the HP driver I'm using.

Possibly there is another external file that the HP stores (maybe an INI, XML, JSON, or similar file) that affects the setting to use. If you can see if one exists, possibly changing the profile file(s) at runtime will work, but I have never tried this.

Let me know if you find anything that affects the profile.

[dynarithmic]

OK, I took another look at the log you produced, and the "HP TWAIN USB" does support CAP_CUSTOMDSDATA and various custom capabilities.

As to the CAP_CUSTOMDSDATA, you can try this:

1) Make sure the profile is selected by issuing the DTWAIN_ShowUIOnly call, and set the profile (you won't be able to scan).

2) See if the settings you were trying before with B/W now take affect when the UI is not displayed.

[dynarithmic]

As to the logs if you search for:

Source "HP TWAIN USB" contains the following 118 capabilities:

You will see the capabilities listed. Note there are many custom caps. I don't know what each one is responsible for, since it's all up to HP what they do.

Even though I have an HP scanner, the caps available come nowhere near the ones available for the HP Scanjet 7000 S3. So me mentioning about the unavailability of custom caps was in regards to my setup. The ScanJet 7000 is much more capable than the one I have (an all-in-one printer/fax/scanner).

It looks like HP uses the same UI skin for all their scanners, which probably makes sense.

[sean-neeley]

OK, I took another look at the log you produced, and the "HP TWAIN USB" does support CAP_CUSTOMDSDATA and various custom capabilities.

As to the CAP_CUSTOMDSDATA, you can try this:

1. Make sure the profile is selected by issuing the DTWAIN_ShowUIOnly call, and set the profile (you won't be able to scan).
2. See if the settings you were trying before with B/W now take affect when the UI is not displayed.

Ok, I played around with this. I found that the scan profiles are stored in C:\Users\xyz\AppData\Local\HP\HP Scan\ScanApp.ini

The black and white settings are in a section:

[TWAIN_PROFILE_2]
ProfileNameENU =Black and White Document

When you change to "faster" black and white mode in the UI, this TWAIN_PROFILE_2 is modified so that several features are disabled. For example ContentDeskew goes from 1 to 0. I also noticed that ColorMode changes from 17 to 16.

The difficulty that I'm having is that each time the HP UI is opened, it switches back to the "Faster Black and White" profile. If I change the profile to normal Black and White, scan, close the dialog, then scan without the UI it still goes back to Faster Black and White. So somehow I need to add code that makes it not do this.

Can you provide more info on how to set custom data values? I wrote code to call DTWAIN_GetCustomDSData and it returns success, but i'm not sure how to take the integer that is returned and turn that into something useful that I can inspect. I can't find any C# examples anywhere. I was hoping to reverse engineer it. I requested developer access on the HP web site to get access to their SDK docs, but I don't know how long it will be until they grant access.

ScanApp - Fast BW.ini.zip

Thanks again.

[sean-neeley]

OK, after spending all day on this, I finally got somewhere. The code we were using was:

TwainAPI.DTWAIN_SetPixelType(selectedSourceParam, TwainAPI.DTWAIN_PT_BW, 1, true);
TwainAPI.DTWAIN_AcquireBufferedEx(
                        dsSelectedSource,
                        TwainAPI.DTWAIN_PT_DEFAULT, /* pixel type */,
                        TwainAPI.DTWAIN_MAXACQUIRE, /* number of pages */
                        FALSE, /* show UI */
                        TRUE, /* close source */
                        acquireArray,
                        ref status
                    );

I commented out DTWAIN_SetPixelType() and instead pass the proper pixel type on AcquireBufferdEx(), like so:

//TwainAPI.DTWAIN_SetPixelType(selectedSourceParam, TwainAPI.DTWAIN_PT_BW, 1, true);
TwainAPI.DTWAIN_AcquireBufferedEx(
                        dsSelectedSource,
                        TwainAPI.DTWAIN_PT_BW, /* pixel type */,
                        TwainAPI.DTWAIN_MAXACQUIRE, /* number of pages */
                        FALSE, /* show UI */
                        TRUE, /* close source */
                        acquireArray,
                        ref status
                    );

This code change makes it so that the "Faster Black/White" Color Mode is no longer selected when I run the program with UI enabled. However, now I have to open the UI at least once before scanning to get Rotation, Contrast, Brightness and other features to do anything. After the UI has been opened once, I can scan without the UI and Rotation, etc work. When restarting the app, I have to open the UI dialog again to get these features to work.

I'm starting to run out of ideas at this point. We are having better luck with grayscale scanning on this HP, so I may just change my program to scan grayscale and then convert to B&W programatically.

[dynarithmic]

Ok, I see a bug in the C# bindings for DTWAIN_GetCustomDSData and DTWAIN_SetCustomDSData.

The C# binding for that function looks like a holdover from an earlier version of DTWAIN.

If you go to the API docs for the function, you see this:

HANDLE DTWAIN_GetCustomDSData (

DTWAIN_SOURCE | Source, -- | -- LPBYTE | lpData, LONG | nSize, LPLONG | pActualSize, LONG | nFlags );

The second parameter should be an array of bytes that will get filled in by the API if the last parameter (flags) is set to DTWAINGCD_COPYDATA.

Basically, that array of bytes returned from DTWAIN_GetCustomDSData is what you should inspect. The second parameter should be marshalled over as an array of bytes to be filled in, not just as a reference to an int.

---

The same issue with the DTWAIN_SetCustomDSData:

HANDLE DTWAIN_SetCustomDSData (

DTWAIN_SOURCE | Source, -- | -- HANDLE | hData, LPBYTE | pData, LONG | nSize, LONG | nFlags );

For DTWAIN_SetCustomDSData, use the DTWAINSCD_USEDATA flag.

If you could/should change the C# dtwain*.cs file(s) to properly put the correct binding in, that would be great. I would have to do this and test the C# workings of the code, so expect a little bit more time for me to do this.

[dynarithmic]

Ok, I quickly tested using the following binding:

        [DllImport(DTWAIN_LIBRARY, CharSet=CharSet.Auto,
        ExactSpelling=true, CallingConvention=CallingConvention.StdCall)]
        public static extern int DTWAIN_GetCustomDSData(DTWAIN_SOURCE Source, byte [] Data, int dSize, ref int pActualSize, int nFlags);

And the following snippet:

DTWAIN_SOURCE SelectedSource = IntPtr.Zero;
//...
SelectedSource = TwainAPI.DTWAIN_SelectSource();
byte[] data = new byte [1000];
int actual = 0;
TwainAPI.DTWAIN_GetCustomDSData(SelectedSource, data, 1000, ref actual, TwainAPI.DTWAINGCD_COPYDATA);

The data is filled in with the custom data.

Note that I arbitrarily chose 1000 as the size. If you want the exact size, you should call the function twice, the first time specifying 0 instead of 1000, then the actual will be populated with the number of bytes. Then you call it a second time with the value of actual instead of 0.

int actual = 0;
TwainAPI.DTWAIN_GetCustomDSData(SelectedSource, null, 0, ref actual, TwainAPI.DTWAINGCD_COPYDATA);
byte[] data = new byte[actual];
TwainAPI.DTWAIN_GetCustomDSData(SelectedSource, data, actual, ref actual, TwainAPI.DTWAINGCD_COPYDATA);

You should make similar changes to DTWAIN_SetCustomDSData.

---

Note: I do not know what format the custom data will be in. It could be in a binary format that would be difficult, if not impossible to interpret. It could be an XML file, JSON, or some other text format where it would be understandable, and probably could be adjusted and sent with a call to DTWAIN_SetCustomDSData. You will know what to do as soon as you are able to test DTWIAN_GetCustomDSData.

[sean-neeley]

Thanks! The byte[] makes a lot more sense. I think you have one mistake though. It should be

    public static extern int DTWAIN_GetCustomDSData(DTWAIN_SOURCE Source, **ref** byte[] Data, int dSize, ref int pActualSize, int nFlags);

instead of:

    public static extern int DTWAIN_GetCustomDSData(DTWAIN_SOURCE Source, byte[] Data, int dSize, ref int pActualSize, int nFlags);

I'll try it out with a scanner next week and report back.

[sean-neeley]

Update: I just received access to the HP SDK documentation. There is a capability to turn off the HP "Fast Black and White" mode, which we have been struggling with.

// Set to TRUE/FALSE to enable/disable faster black and white mode
#define CAP_HP_FAST_BW_ENABLED              (CAP_CUSTOMBASE + 0x50)

I will be experimenting with this soon. It looks very promising.

[dynarithmic]

Ok, in the logs you originally posted there is CAP_CUSTOMBASE + 80, which is the one you're looking for. I am assuming it is a TW_BOOL type, where you either give it a 1 or 0 to turn on/off the B/W mode.

Probably something like:

DTWAIN_ARRAY array_ = TwainAPI.DTWAIN_ArrayCreateFromCap(Source, TwainAPI.DTWAIN_CV_CAPCUSTOMBASE + 80, 1);
TwainAPI.DTWAIN_ArraySetAtLong(array_, 0, 0);  // The last parameter is 0 == FALSE, and we assume it will hold int (LONG/BOOL) type
TwainAPI.DTWAIN_SetCapValues(Source, TwainAPI.DTWAIN_CV_CAPCUSTOMBASE + 80, TwainAPI.DTWAIN_CAPSET, array_);

Of course, you want to check the return value for each of these functions to test if they all return OK.

[sean-neeley]

Yep, there is some sample code from HP and TW_BOOL is correct. However, the DTWAIN docs say to use DTWAIN_SetCapValuesEx2() for custom capabilities. Is that not true?

This was the code I started writing before I saw your answer:

                DTWAIN_ARRAY theValues = TwainAPI.DTWAIN_ArrayCreate(TwainAPI.DTWAIN_ARRAYLONG, 1);
                TwainAPI.DTWAIN_ArraySetAtLong(theValues, 0, FALSE);
                if (TwainAPI.DTWAIN_SetCapValuesEx2(selectedSourceParam, HPTwain.CAP_HP_FAST_BW_ENABLED,
                    TwainAPI.DTWAIN_CAPSET, TwainAPI.DTWAIN_CONTONEVALUE, TwainAPI.DTWAIN_TWTY_BOOL, theValues) == TRUE)
                {
                    // TODO - log success
                }
                else 
                {
                    // TODO - log failure
                }
                TwainAPI.DTWAIN_ArrayDestroy(theValues);

[dynarithmic]

Yes, you should use the DTWAIN_SetCapValuesEx2 function, provided you have all the information about the container type and data type.

Actually DTWAIN attempts to figure out the data type and container when the source is first opened. However for custom caps, it is always safer to get the information from the manufacturer's documentation.

[sean-neeley]

I tried both your code block and my own and both worked. The call to disable fast black & white mode only succeeds if called after DTWAIN_SetPixelType(). I think I'm good to go at this point. I appreciate all of your help.