Is this library usable from Python bindings?

[devinus]

I'm wondering what a basic Python bindings would look like. It looks like there was already thought put into this, but any direction would be very appreciated.

[dgoldman-pdx]

@devinus the existing references to Python in this DMZ code are there because we use the DMZ when we work on building improved neural network models for identifying card numbers and expiry dates. This work involves Cython, which is probably a more complex approach than what you need. But some of the declarations under #if CYTHON_DMZ might make good models.

The main entry point to the library that you would need to support is this function:

void scanner_add_frame_with_expiry(ScannerState *state, IplImage *y, bool scan_expiry, FrameScanResult *result);

which is declared in scan.h. This function takes a grayscale image (428 x 270 pixels) and returns its best guess of card number and expiry date in state and result.

You can see this function in use in our iOS or Android libraries.

To obtain the 428 x 270 card image, use these functions:

// Convenience method that returns whether a set of found_edges contains all edges as being found.
bool dmz_found_all_edges(dmz_edges found_edges);

// Detect card edges, and calculate the corner points if all four edges have been detected.
// The boolean return value indicates whether a card was successfully detected.
bool dmz_detect_edges(IplImage *y_sample, IplImage *cb_sample, IplImage *cr_sample,
                                       FrameOrientation orientation, dmz_edges *found_edges, dmz_corner_points *corner_points);

// Convert a sample from the camera to a transformed, rectified card image.
// Use corner_points from dmz_detect_edges.
// *transformed MUST be initialized to NULL or a valid IplImage. It is the caller's responsibility
// to free transformed.
void dmz_transform_card(dmz_context *dmz, IplImage *sample, dmz_corner_points corner_points, FrameOrientation orientation, bool upsample, IplImage **transformed);

which are declared in dmz.h.

I know that's only the first sketch of an answer to your question. Feel free to continue the discussion as you peruse the code.

[josharian]

Note that the dmz is designed to be small, fast, and imprecise, because it runs on a tiny processor. We compensate for the imprecision by feeding the scanner many frames and aggregating the results.

This is a good trade-off on the device, since frames are cheap (the camera spits them out at >=30 hertz) and computation is expensive. It is not a good trade-off on a server, where frames are expensive (they must go over the network) and computation is cheap.

Code aside, I would discourage you from doing this work on a server. Credit card images contain very sensitive information. In particular, for PCI compliance, the cvv must never be stored, and the cvv is on the front of some credit cards, notably amex. Also, users are understandably sensitive to what goes over the network. Keeping processing purely on-device and transient is better all around.

[devinus]

@dgoldman-ebay This is perfect, thank you. I already started going down the path of just using Cython yesterday here: https://github.com/devinus/cardio

I was thrashing around the JNI source trying to figure everything out. This may seem like a dumb question, but what does DMZ stand for and what should the dmz_context actually hold?

@josharian For my purposes, credit card images will be transferred over HTTPS and I never intend to store anything on the server. Completely transient. The reason I want to use card.io is because it's already so well tested and I couldn't find good solutions anywhere else that didn't require me knowing computer vision.

[josharian]

dmz stands for "demilitarized zone" -- code that is not platform-specific. Although some platform specific code snuck in, you'll note that it is in files called mz. :) dmz_context currently to allow platforms that to associate data as needed.

@dgoldman-ebay is all our Cython code bound up in training, etc? Is there anything standalone that we could share (if just informally) to help? My recollection is that it took me days of floundering and suffering to get all the Cython layers set up correctly originally.

[devinus]

What are the y_sample, cb_sample, and cr_sample IplImages supposed to represent in dmz_detect_edges?

[devinus]

I'm guess I need to use dmz_deinterleave_uint8_c2 to derive cb_sample and cr_sample from reading https://github.com/card-io/card.io-Android-source/blob/master/card.io/src/main/jni/nativeRecognizer.cpp#L306

[josharian]

Most mobile camera provide biplanar YCbCr output, with Cb and Cr interleaved and at half resolution. As you found, dmz_detect_edges and the rest of the dmz assumes separate y, cb, and cr planes. Given an interleaved cb and cr plane, dmz_deinterleave will separate the planes.

I'm more than happy to answer more questions as you have them. If you wanted to return the favor, you could send some PRs turning the answers into concise, localized docs. :)

[dgoldman-pdx]

@dgoldman-ebay is all our Cython code bound up in training, etc? Is there anything standalone that we could share (if just informally) to help? My recollection is that it took me days of floundering and suffering to get all the Cython layers set up correctly originally.

I've just added a cython_dmz subdirectory to this repo. This is a replica of the directory we use when building and running our own Python model-training scripts using Cython. There's a README.md in there with a few sketchy hints and remarks.

@devinus feel free to continue to ask for clarification about any of this!

[devinus]

Hey guys, I'm coming to the pool of knowledge again.

I've been trying to get this to work by copying the Android SDK's scan method here: https://github.com/devinus/cardio/blob/master/cardio.pyx#L36

But I cannot seem to scan expiry no matter what image I use (see: creditcard.bmp, creditcard2.bmp, creditcard3.bmp). Am I doing something totally wrong?

The output right now is this:

Image size: (640, 480)
Focus score: 31.6824798584
Card detected: True
Found all edges: True
Usable: True
Upside down: False
vseg score: 26.4310188293
Scan complete: False
# of numbers: 16
Expiry month: 1546028640
Expiry year: 32767
[1, 2, 3, 4, 5, 6, 7, 8, 9, 8, 7, 6, 5, 4, 3, 2]

Edit: Got numbers to scan, changed question to expiry

[devinus]

I feel like I'm missing something simple but important to scan expiry correctly.

[devinus]

This is where I pretend to scan 3 different frames (but they're the same image): https://github.com/devinus/cardio/blob/master/cardio.pyx#L78

[josharian]

It's possible that you're missing something -- @dgoldman would be the right person to answer that question -- but it is also possible that the expiry scanner simply fails on that particular image. Expiry scanning is not nearly as reliable as card scanning right now. You might want to start by trying a few other images, perhaps of other cards.

[devinus]

@josharian I'm testing against 4 different card images right now and they all fail. Is it perhaps that unreliable?

[devinus]

Oh, and these are the card images I'm testing against:

https://github.com/devinus/cardio/blob/master/creditcard.bmp https://github.com/devinus/cardio/blob/master/creditcard2.bmp https://github.com/devinus/cardio/blob/master/creditcard3.bmp

[josharian]

creditcard2.bmp is the only one I'd expect to scan; the others are not typical embossed credit cards, which is the primary target. I don't know what the expiry success rate is in the wild, but failing on one or two is probably expected at this point. I'll let @dgoldman-ebay take it from here, though. He can say more definitively.

[devinus]

Okay, great. Thankfully they all scan the numbers. The only one I haven't been able to scan had gold numbers.

[dgoldman-pdx]

@devinus consistent with what @josharian indicated, the font used in the first and third sample images is quite different from the typical font on which we have trained our digit recognizers. I'm actually impressed that the card-number recognizer is apparently handling them!

At this point, expiry recognition is definitely hit-or-miss depending on the card. Although to my eye, the expiry on your creditcard2 should probably be scannable. (Note: the human eye is an unreliable tool for making such a guess!)

Our code requires several successful scans of the card number and of the expiry before it concludes that it has arrived at a reasonable guess of each. It's possible that if you submit that same image more than 3 times (6 times? 10 times?) you'll get a successful expiry scan.

If you use a DEBUG version of the dmz code, then you should see a lot of additional information in your console logs that will help indicate what's happening during the expiry segmentation and categorization steps.

[dgoldman-pdx]

@devinus feel free to reopen this issue, or start a new issue, if you're still working on this and you have any further questions, observations, or suggestions!

[devinus]

@dgoldman-ebay Yep, definitely still working on it! I'm successfully using card.io from Python. I want to improve DMZ itself to supprot a STANDALONE_DMZ option, because I've only been using Cython to prototype this. I'm going to create a native Node.js library that can use card.io soon.

[dgoldman-pdx]

:+1:

[jinglejiang]

@devinus Hi Devis I am recently trying to do similar things as you did - cythonize cardio so it can be callable from python. I find your repo https://github.com/devinus/cardio but run into some cryptic errors. can you help?

Specifically, after set up all submodule and stuff, I make

then it returns:

c++ -DCYTHON_DMZ -DSCAN_EXPIRY -Idmz -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7/pkg-config --libs --cflags opencv-shared dmz/dmz_all.cpp -o libdmz.so Undefined symbols for architecture x86_64: "cv::Exception::Exception(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)", referenced from: llcv_hough(void const*, _IplImage*, _IplImage*, float, float, int, float, float, bool, float) in dmz_all-b3f6d4.o llcv_canny7_precomputed_sobel(_IplImage*, _IplImage*, _IplImage*, _IplImage*, double, double) in dmz_all-b3f6d4.o llcv_equalize_hist(_IplImage const*, _IplImage*) in dmz_all-b3f6d4.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [libdmz.so] Error 1

If I run python setup.py build_ext --inplace seperately, then it errors out with ld: library not found for -ldmz clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command 'clang++' failed with exit status 1

Do you have any suggestions? Thanks in advance!!

[dgoldman-pdx]

@jinglejiang I'm not much involved with this project anymore. But your error messages are so familiar to me that I can at least help this time!

See https://github.com/card-io/card.io-iOS-SDK/issues/93

[jinglejiang]

Hi @dgoldman-ebay thanks for your reply! Unfortunately that post does not quite help my case as I am not building cardio-ios-source neither am I using xcode....

However according to the idea of that post, I did try to change @devinus's setup.py with further flag of '-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7' but it stills fail on make or python setup.py build_ext --inplace. The specific error of failure is ld: library not found for -ldmz clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command 'clang++' failed with exit status 1

At this time I believe really only @devinus could come to rescue as he developed this cython code.... Can you plz help? My best guess is this setup.py missing something. I am using macOS.