CHARnn fields gets cut off after the first zero byte

clausreinke commented 3 years ago

We're currently experiencing a problem with getting data from a search help. RECORD_TAB result entries are defined as CHAR3000 and may contain non-printable characters, the data we are looking for at the moment is placed after those characters.

our old JCO-based app has no trouble getting the data
if we enable SAPNWRFC tracing (level 4), we see the complete entries as expected, so the C connector has no trouble with the data
if we try to log the entries after client.invoke, the string is truncated before the non-printable characters, so we have no access to the part of the data we need

We cannot find any restrictions on ABAP type c, other than UCS-2, neither the Java nor the C connector seem to have problems with this data, and JS can handle non-printable characters in strings just fine, so we think this may be a bug in node-rfc, somewhere between C connector and JS.

The search help is non-standard, so we do not have a reproducable example. Below is a sample entry, as copied from the SAPNWRFC trace (passed through od -v -t x1z): the value we get from node-rfc is truncated after TESTUSER3, which when trimmed is the last text before the non-printable characters 00 00 00 0c; the data we need would be the 1000329, which would be the the first text after the non-printable characters.

1027140 3e 20 52 66 63 53 74 72 75 63 74 75 72 65 20 6e  >> RfcStructure n<
1027160 61 6d 65 3d 22 2f 53 53 43 2f 54 42 57 46 5f 57  >ame="/SSC/TBWF_W<
1027200 45 42 5f 50 41 52 41 4d 45 54 45 52 22 0d 0a 09  >EB_PARAMETER"...<
1027220 57 69 64 74 68 3a 20 36 30 30 30 09 09 46 69 65  >Width: 6000..Fie<
1027240 6c 64 43 6f 75 6e 74 3a 20 31 0d 0a 09 46 69 65  >ldCount: 1...Fie<
1027260 6c 64 4e 61 6d 65 3a 20 46 45 4c 44 09 09 56 61  >ldName: FELD..Va<
1027300 6c 75 65 3a 20 20 20 20 31 30 30 30 32 32 30 20  >lue:    1000220 <
1027320 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027340 20 20 20 20 20 20 20 20 20 20 20 54 65 73 74 20  >           Test <
1027360 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027400 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027420 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027440 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027460 20 54 65 73 74 33 20 20 20 20 20 20 20 20 20 20  > Test3          <
1027500 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027520 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027540 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027560 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027600 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027620 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027640 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027660 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027700 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027720 20 20 20 54 45 53 54 55 53 45 52 33 20 20 20 20  >   TESTUSER3    <
1027740 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1027760 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030000 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030040 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030060 20 20 20 20 20 20 20 20 20 20 20 20 00 00 00 0c  >            ....<
1030100 20 20 20 20 20 31 30 30 30 33 32 39 20 20 20 20  >     1000329    <
1030120 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030140 20 20 20 20 20 20 20 20 32 30 31 37 30 37 31 32  >        20170712<
1030160 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030200 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030220 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
1030240 20 20 20 30 30 30 30 30 30 30 30 0d 0a 3c 3c 20  >   00000000..<< <
1030260 52 66 63 53 74 72 75 63 74 75 72 65 0d 0a 0d 0a  >RfcStructure....<

That corresponds to what we see in the ABAP debugger for this field around the non-printable characters:

bsrdjan commented 3 years ago

Which SAP NWRFC SDK release is used? Did it work before, with older SDK release?

clausreinke commented 3 years ago

$ node -p "require('node-rfc').environment"
{
  platform: { name: 'win32', arch: 'x64', release: '10.0.19042' },
  env: {
    SAPNWRFC_HOME: 'C:\\<project>\\dependencies\\nwrfc750P_6-70002755\\nwrfcsdk',
    RFC_INI: '',
    nwrfcsdk_lib_on_path: true
  },
  versions: {
    node: '14.15.2',
    v8: '8.4.371.19-node.17',
    uv: '1.40.0',
    zlib: '1.2.11',
    brotli: '1.0.9',
    ares: '1.16.1',
    modules: '83',
    nghttp2: '1.41.0',
    napi: '7',
    llhttp: '2.1.3',
    openssl: '1.1.1g',
    cldr: '37.0',
    icu: '67.1',
    tz: '2020a',
    unicode: '13.0'
  },
  noderfc: {
    version: '2.3.1',
    nwrfcsdk: { major: 7500, minor: 0, patchLevel: 6 }
  }
}

No, this never worked for us with node-rfc, we just thought there was a bug in our code that caused the FM to behave differently when called from our two apps - we were literally looking in all the wrong places before we stumbled on this issue.

bsrdjan commented 3 years ago

The bug was caused by casting SDK CHAR fields to c-strings, before conversion to node strings. CHAR strings containing zero bytes in-between were therefore terminated after the first zero byte found from left.

It is now fixed in main branch b4aeaf9, by using std::strings: nwrfcsdk.cc#L421 instead.

You can build the main branch from source or use this pre-built and tested Windows binary, with this fix included:

node-rfc-v2.3.1-napi-v7-win32-x64.tar.gz

The fix does not work with PL7 and will be included in next node-rfc release, soon after the SDK PL8 shipped.

After the fix d7f516b, NodeJS strings sent to ABAP CHARnn fields are not terminated after the first zero byte. ABAP statements expecting unicode strings, might not work as expected with such strings.

clausreinke commented 3 years ago

Thanks for the fast turnaround! We're not yet on the latest node-rfc version (why does an rfc client need RFC authorization for function module "SYSTEM_RESET_RFC_SERVER" when there is no call to it?), but I've backported your fix to our version and it seems to do the trick for this issue with SDK PL6. So we'll just need to skip SDK PL7 and go straight to PL8 when it comes out?

bsrdjan commented 3 years ago

The 2.4.0 release with the fix is published, feel free to test. There is no public release plan for PL8 but I expect the release dynamic similar to past years.

SYSTEM_RESET_RFC_SERVER is called when RfcResetServerContext( ) SDK API is called. It is called by Pool connections internal recycling logic and also exposed as node-rfc resetServerContext method, for node-rfc client applications.

Here more info on connector applications authorisations: https://launchpad.support.sap.com/#/notes/460089

bsrdjan commented 3 years ago

Apparently fixed, please re-open if needed.

SAP / node-rfc

CHARnn fields gets cut off after the first zero byte #193