Built-in NFC reader is truncating responses

dainnilsson commented 3 years ago

Hello! I have a rather strange (too me at least) issue I'm running into.

I have a Lenovo Carbon X1 laptop with a built in NFC reader. I'm running Windows 10. python-pyscard lists the reader as: ['Microsoft IFD 0', 'NXP NXP's Proximity based PCSC Reader 0']

I'm not sure why it gets 2 entries, but the first one (Microsoft) is the one that seems to somewhat work. I can connect to it and transmit commands. The second one (NXP) never shows as having a card connected.

Here's where I'm running into trouble: Response APDUs seem to get truncated to 8 bytes, where pyscard treats the first 6 as data, and the last 2 as SW1, SW2.

Here's an example to illustrate this. Using a different reader (getting a correct reponse): connection.transmit([0, 164, 4, 0, 8, 160, 0, 0, 5, 39, 71, 17, 23]) out: ([86, 105, 114, 116, 117, 97, 108, 32, 109, 103, 114, 32, 45, 32, 70, 87, 32, 118, 101, 114, 115, 105, 111, 110, 32, 53, 46, 50, 46, 54], 144, 0)

Using the built-in reader (getting a truncated response): connection.transmit([0, 164, 4, 0, 8, 160, 0, 0, 5, 39, 71, 17, 23]) out: ([86, 105, 114, 116, 117, 97], 108, 32)

Now to add to my confusion, I've also found that:

A C program using PCSC doesn't have this issue, the built-in reader works fine.
Attaching the built-in reader to a VMWare Guest running Ubuntu, and running python-pyscard inside the guest against it also works fine.

Any clues as to what might be going on here? Anything I can provide to help figure this out? Thankful for any suggestions!

LudovicRousseau commented 3 years ago

What version of pyscard?

dainnilsson commented 3 years ago

This test now was running pyscard 1.9.9 (under Python 3.8.5).

I should also note that I haven't tested older versions, so I don't know if this particular reader has ever worked or not.

LudovicRousseau commented 3 years ago

Very strange bug.

PC/SC works fine since a C program does not have the issue
pyscard works fine since it does work with another NFC reader

Getting debug information at the PC/SC level could help. But I don't know how to do that on Windows.

jacopo-j commented 2 years ago

I'm experiencing the same bug on a Lenovo ThinkPad T14 Gen1, running Python 3.9.2 and pyscard 2.0.1 (Windows 10). I'm also using the built-in NFC reader "Microsoft IFD 0" and I'm not experiencing the bug using external readers. Futhermore, I do not experience the bug using PCSC libraries other than pyscard (for example, node-pcsc).

Is there anything I can do to help troubleshoot the issue?

LudovicRousseau commented 2 years ago

I suggest to use a PC/SC sniffer to know if PySCard received the correct answer but misused it itself or if the problem comes from a lower level layer. Maybe you can use https://tech.springcard.com/2016/scardsniffer-pcsc-spy/ or a similar tool.

LudovicRousseau commented 2 years ago

Another (simpler) option is to apply the patch to add some debug:

diff --git a/smartcard/scard/scard.i b/smartcard/scard/scard.i
index d04731e..51b57df 100644
--- a/smartcard/scard/scard.i
+++ b/smartcard/scard/scard.i
@@ -698,6 +698,7 @@ static SCARDRETCODE _Transmit(
             return SCARD_E_INVALID_PARAMETER;

     }
+    printf("cBytes in: %d\n", pblRecvBuffer->cBytes);
     ret = (mySCardTransmit)(
                 hcard,
                 piorequest,
@@ -706,6 +707,7 @@ static SCARDRETCODE _Transmit(
                 NULL,
                 pblRecvBuffer->ab,
                 &pblRecvBuffer->cBytes);
+    printf("cBytes out: %d\n", pblRecvBuffer->cBytes);

     return ret;
 }

Rebuild pyscard, reinstall it and then execute your sample code again and send me the output.

jacopo-j commented 2 years ago

Stupid question: where should I get the output after installing the patched library?

I tried using PCSC sniffers but neither the one you suggested nor https://www.mysmartlogon.com/knowledge-base/trace-apdu-on-windows/ seem to work (maybe they're outdated?)

LudovicRousseau commented 2 years ago

Sorry. You execute your code that exhibits the problem and you should see the 2 printf() messages on the console.

jacopo-j commented 2 years ago

I asked because I couldn't see anything at my first attempt. But I tried again and this time it worked; probably I didn't remove the original module properly.

Anyway, here is the result:

>>> conn.transmit(toBytes("00 A4 04 00 0E 32 50 41 59 2E 53 59 53 2E 44 44 46 30 31 00"))
cBytes in: 65544
cBytes out: 8
([111, 54, 132, 14, 50, 80], 65, 89)
>>>

The correct expected output from the smart card starts with those 8 bytes (6 data bytes + those incorrectly interpreted as SW1 and SW2) but is 56 byte long (58 including SW1 and SW2).

LudovicRousseau commented 2 years ago

cBytes out: 8 indicates that the Windows PC/SC layer returned a length of 8 bytes. PySCard never received the other bytes.

Can you write a C program to see if you have the same problem? @dainnilsson said the problem is not present for a C program but I am very surprised by that. You can have a look at https://ludovicrousseau.blogspot.com/2010/04/pcsc-sample-in-c.html to get an idea of how to write a PC/SC program in C.

jacopo-j commented 2 years ago

I wrote a C program using the sample code you linked and the same APDU command I used in earlier tests. I can confirm that this way the response is correct and complete.

LudovicRousseau commented 2 years ago

Thanks @jacopo-j Now can you write a Python program but using the low level API. The Python code should be very similar to the C version. Have a look at https://ludovicrousseau.blogspot.com/2010/04/pcsc-sample-in-python.html

jacopo-j commented 2 years ago

The response is truncated using the low level API. So I guess the bug must be somewhere in the Python wrapper

LudovicRousseau commented 2 years ago

Maybe you are right. But with the patch above you see the length value returned by PC/SC. So even before PyCSCard can truncate the data.

What is the card ATR? Can you post the C and low level Python programs you used?

jacopo-j commented 2 years ago

Here you go. There are very minor changes with respect to your example code.

I did all the tests with several cards, the one I have here has ATR 3B 80 80 01 01. (The ATR seems to be always returned correctly, even if longer than 8 bytes).

LudovicRousseau commented 2 years ago

The card supports both T=0 and T=1. I was suspecting a difference in the SCardConnect()call but in both cases you use SCARD_PROTOCOL_T0 | SCARD_PROTOCOL_T1.

It is very strange. It would really help if you could use a PC/SC spy like I mentioned in a previous message. Use the spy in the 2 cases: C and Python. Another idea is to use a USB spy. Maybe something is different.

jacopo-j commented 2 years ago

I'll do my best to make PC/SC sniffing work in the next days; I'll let you know if I succeed or not. USB sniffing is not applicable because the reader we are talking about is internal and connected via I2C.

jacopo-j commented 2 years ago

I was able to spy PC/SC traffic by using API Monitor with additional definitions as explained here: https://www.mysmartlogon.com/knowledge-base/trace-apdu-on-windows/. That basically allowed me to capture all calls to winscard.dll functions with their parameters and return values.

The result is that SCardTransmit returns a truncated 8 byte response (with length = 8) when called using my test Python program, and the complete 58 byte response when called using my test C program.

Unfortunately API Monitor only allows to export captures in a proprietary format, so you need a Windows machine to view them. Let me know if you want them -- if possible, I'd prefer to send them to you in private as I'm not 100% sure they don't contain sensitive/personal data. Or maybe I can share some screenshots (not the best thing in the world...)

Finally, I'll share some differences I spotted between the C test and the Python test, ~~I don't know whether they are relevant or not:~~ (Edit: I tried changing both of them and they didn't seem to make any difference)

My test program runs in 32 bit, Python runs in 64 bit, so they technically use two different versions of the Winscard library
The C program calls SCardEstablishContext with SCARD_SCOPE_SYSTEM, while the Python program uses SCARD_SCOPE_USER

LudovicRousseau commented 2 years ago

Thanks for the tests. You can email me the proprietary files at ludovic.rousseau at free.fr

Another important function is SCardConnect().

LudovicRousseau commented 2 years ago

@jacopo-j I installed API Monitor and analysed you 2 traces.

In the Python execution I see a "Alcor Micro" reader returned by SCardLIstReadersA(). I guess it is an external USB reader that was connected at that time. It should not be a problem since in the 2 cases you use the same reader "Microsoft IFD 0".
if I understand correctly you regenerated your C sample to be compiled in 64bits instead of 32bits and the problem is still present. Exact?
in your C sample you used SCARD_SCOPE_USER instead of SCARD_SCOPE_SYSTEM(that should not change anything) and the problem is still present. Exact?

jacopo-j commented 2 years ago

Correct. If there is any possibility that the other reader may interfere I can try again without it, I didn't think it could be a problem.
Correct. More exactly, it still works, since the C version never had problems.
I actually did the other way around, tried SCARD_SCOPE_SYSTEM in the Python version, and it did not solve the problem.

LudovicRousseau commented 2 years ago

In the C sample trace I see in the call trace that the program uses kernel32.dll and ntdll.dll. But I do not see any reference to winscard.dll. winscard.dll is loaded by the Python wrapper at https://github.com/LudovicRousseau/pyscard/blob/master/smartcard/scard/winscarddll.c#L658 Could that be a problem?

Maybe some component in the stack (Windows, winscard, the reader driver, etc.) has a check to fail when used by a Python program? Maybe you can try to install a clean Windows in a Virtual Machine and see what happens? @dainnilsson already used a GNU/Linux inside a VM and it worked as expected. Do the same test with Windows instead.

This bug is very strange. It is the first time I see something like that. I don't know what is special with these Lenovo laptops.

LudovicRousseau commented 2 years ago

Try this patch:

diff --git a/smartcard/scard/scard.i b/smartcard/scard/scard.i
index d04731e..fd00bb5 100644
--- a/smartcard/scard/scard.i
+++ b/smartcard/scard/scard.i
@@ -154,7 +154,7 @@ Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
 // MAX_BUFFER_SIZE_EXTENDED is pcsc-lite specific
 // Windows 7 does not support more than 65544 bytes for SCardControl()
 // See https://github.com/LudovicRousseau/pyscard/issues/19
-#define MAX_BUFFER_SIZE_EXTENDED   65544 
+#define MAX_BUFFER_SIZE_EXTENDED   258 
 #endif //PCSCLITE

 #include "pcsctypes.h"

jacopo-j commented 2 years ago

The patch worked!

LudovicRousseau commented 2 years ago

OK. Now modify your C sample code to increase the receiving buffer used in SCardTransmit(). It is pbRecvBuffer in https://ludovicrousseau.blogspot.com/2010/04/pcsc-sample-in-c.html

Replace BYTE pbRecvBuffer[258]; by BYTE pbRecvBuffer[65544]; and you should reproduce the bug.

jacopo-j commented 2 years ago

Yes, I could reproduce it

LudovicRousseau commented 2 years ago

Now I propose to report the bug to Lenovo.

You can also try to find the maximal size that does not generate the bug so I can update PySCard using this value.

jacopo-j commented 2 years ago

The maximum size that works is 65535. Hopefully that won't make much of a difference for all other users.

LudovicRousseau commented 2 years ago

The previous value 65544 is 0x10008. The output was truncated at 8 bytes. I guess only the low 16bits of the length were used.

Stupid Windows developers!

dainnilsson commented 2 years ago

I can confirm the latest release (2.0.2) resolved the issue on the Thinkpad X1 Carbon I was initially testing this on as well. Thanks to both of you, great job on troubleshooting and fixing this!

LudovicRousseau / pyscard

Built-in NFC reader is truncating responses #100