Open nprobert opened 3 years ago
Yes, UPER (and OER) encoding seems to have some problems.
Would you mind uploading the simplest/shortest ASN.1 file and XER-encoded data that manifest this problem?
All those mallocs make me nervous. I ran valgrind on the check programs, got nothing. So I wrote a test program to call my routine above and ran valgrind on it.
==1352920== HEAP SUMMARY:
==1352920== in use at exit: 832 bytes in 24 blocks
==1352920== total heap usage: 49 allocs, 25 frees, 6,599 bytes allocated
==1352920==
==1352920== 256 bytes in 8 blocks are definitely lost in loss record 1 of 3
==1352920== at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920== by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920== by 0x14C048: asn_set_add (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x137A82: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x13765A: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x137539: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==
==1352920== 576 (256 direct, 320 indirect) bytes in 8 blocks are definitely lost in loss record 3 of 3
==1352920== at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920== by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920== by 0x14C048: asn_set_add (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x1379E9: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x13765A: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== by 0x137539: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==
==1352920== LEAK SUMMARY:
==1352920== definitely lost: 512 bytes in 16 blocks
==1352920== indirectly lost: 320 bytes in 8 blocks
==1352920== possibly lost: 0 bytes in 0 blocks
==1352920== still reachable: 0 bytes in 0 blocks
==1352920== suppressed: 0 bytes in 0 blocks
==1352920==
==1352920== For lists of detected and suppressed errors, rerun with: -s
==1352920== ERROR SUMMARY: 98 errors from 6 contexts (suppressed: 0 from 0)
I rewrote the code a bit better, I hope. The API is barely documented here. A simpler valgrind output still shows problems, but within OCTET_STRING_new_fromBuf().
int j2735_encode(const char *raw, int size, char *buf, int max)
{
// assert
if (!raw || !buf)
return -1;
MessageFrame_t *frame = (MessageFrame_t *)calloc(sizeof(MessageFrame_t), 1);
// test
if (!frame) {
return -1;
}
RTCMcorrections_t *rtcm = &frame->value.choice.RTCMcorrections;
// clip
if (size > 1023)
size = 1023;
printf("J2735 Encoded: in=%d, max=%d\n", size, max);
// build frame
frame->messageId = 28; // RTCM
frame->value.present = MessageFrame__value_PR_RTCMcorrections;
// build message
RTCMmessageList_t *list = (RTCMmessageList_t *)calloc(sizeof(RTCMmessageList_t), 1);
if (!list) {
free(frame);
return -1;
}
RTCMmessage_t *oct = OCTET_STRING_new_fromBuf(&asn_DEF_RTCMmessage, raw, size);
RTCMmessage_t *oct = (RTCMmessage_t *)calloc(sizeof(RTCMmessage_t), 1);
if (!oct) {
free(list);
free(frame);
return -1;
}
ASN_SEQUENCE_ADD(&list->list, oct);
// build correction
rtcm->msgCnt = msg_count++;
if (msg_count>=128)
msg_count = 1;
rtcm->rev = RTCM_Revision_rtcmRev2;
rtcm->timeStamp = NULL;
rtcm->anchorPoint = NULL;
rtcm->rtcmHeader = NULL;
rtcm->regional = NULL;
ASN_SEQUENCE_ADD(&rtcm->msgs, list);
// encode
asn_enc_rval_t rc;
rc = uper_encode_to_buffer(&asn_DEF_MessageFrame, NULL, &frame, buf, max);
printf("J2735 Encoded: out=%ld\n", rc.encoded);
ASN_STRUCT_FREE(asn_DEF_MessageFrame, frame);
// check
if (rc.encoded == -1)
return -1;
return rc.encoded;
}
==1397905== Memcheck, a memory error detector
==1397905== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1397905== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1397905== Command: ./test_encode
==1397905==
J2735 Encoded: in=128, max=1398
J2735 Encoded: out=-1
Encoded (128) = -1
==1397905==
==1397905== HEAP SUMMARY:
==1397905== in use at exit: 169 bytes in 2 blocks
==1397905== total heap usage: 8 allocs, 6 frees, 1,809 bytes allocated
==1397905==
==1397905== 169 (40 direct, 129 indirect) bytes in 1 blocks are definitely lost in loss record 2 of 2
==1397905== at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1397905== by 0x13823E: OCTET_STRING_new_fromBuf (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905== by 0x137815: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905== by 0x137632: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905== by 0x137523: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905==
==1397905== LEAK SUMMARY:
==1397905== definitely lost: 40 bytes in 1 blocks
==1397905== indirectly lost: 129 bytes in 1 blocks
==1397905== possibly lost: 0 bytes in 0 blocks
==1397905== still reachable: 0 bytes in 0 blocks
==1397905== suppressed: 0 bytes in 0 blocks
==1397905==
==1397905== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Added ASN_STRUCT_FREE(asn_DEF_OCTET_STRING, oct); near the bottom and got no valgrind errors which kinda tells me that the OCTET_STRING didn't get added to the RTCMmessageList_t structure by ASN_SEQUENCE_ADD somehow.
Traced this back to the asn_set_add() function. Added some debugging code in there and it started working. Removed it and it failed again. Wtf!? Either way, the test message still doesn't get encoded.
calloc(sizeof(RTCMmessageList_t), 1)
should be
calloc(1, sizeof(RTCMmessageList_t))
I'm sorry, I don't have a clear idea of what you're doing, and SAE J2735-2016 "at large" is way too huge for me to comprehend or play with.
I say again - would you be able to create and upload a simple reproducer of the problem? That would include a comprehensible small ASN.1 file and an XER example of encoded data?
J2735 really challenges this compiler, breaks it bad probably. Seems it can't even encode an constrained INTEGER (long) in UPER. The value is supposed to be 12, but looks like a pointer is f'd up in INTEGER_uper.c.
About to encode DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/constr_SEQUENCE_uper.c:379) Encoding MessageFrame->messageId:DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/constr_SEQUENCE_uper.c:402) Encoding NativeInteger DSRCmsgID 78202640 (UPER) (/home/neal/ProbeStar/GitHub/libj2735v2016/src/NativeInteger_uper.c:60) Value 78202640 (04/4) lb 0 ub 32767 ext (/home/neal/ProbeStar/GitHub/libj2735v2016/src/INTEGER_uper.c:178) Failed to encode element DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/INTEGER_uper.c:188)
Memory allocation is badly broken in ways not even valgrind can find. So I just built up my J2735 message by avoiding any dynamic memory allocation. And voila, things work now.
//
// J2735 Encode
//
#include "MessageFrame.h"
static int msg_count = 1;
int j2735_encode(const char *raw, int size, char *buf, int max)
{
// assert
if (!raw || !buf)
return -1;
// clip
if (size > 1023)
size = 1023;
// storage
MessageFrame_t frame;
memset(&frame, 0, sizeof(frame));
RTCMmessageList_t list;
memset(&list, 0, sizeof(list));
RTCMmessage_t msg;
memset(&msg, 0, sizeof(msg));
// frame
frame.messageId = DSRCmsgID_rtcmCorrections_D; // RTCM
frame.value.present = MessageFrame__value_PR_RTCMcorrections;
// corrections
RTCMcorrections_t *rtcm = &frame.value.choice.RTCMcorrections;
rtcm->msgCnt = msg_count++;
if (msg_count>=128)
msg_count = 1;
rtcm->rev = RTCM_Revision_rtcmRev2;
// message list
rtcm->msgs.list.count = 1;
rtcm->msgs.list.size = 1;
// message
msg.buf = (void *)raw;
msg.size = size;
RTCMmessage_t *pmsg = &msg;
rtcm->msgs.list.array = &pmsg;
list.list.array = &pmsg;
list.list.count = 1;
list.list.size = 1;
#ifdef CONSTRAINT_CHECKING
// walk
char err[200];
size_t len = sizeof(err);
if (asn_check_constraints(&asn_DEF_MessageFrame, &frame, err, &len) != 0) {
fprintf(stderr, "%s\n", err);
return -1;
}
#endif
// encode
asn_enc_rval_t rc;
rc = asn_encode_to_buffer(0, ATS_UNALIGNED_CANONICAL_PER, &asn_DEF_MessageFrame, &frame, buf, max);
// check
if (rc.encoded == -1)
return -1;
else if (rc.encoded > max)
fprintf(stderr, "Encoding buffer is too small\n");
return rc.encoded;
}
Encoding RTCMmessage into 17179869185 units of 8 bits (1..1023, effective 10) (OCTET_STRING_uper.c:252)
@nprobert this looks like OCTET_STRING_uper.c
code is getting a pointer that it mis-interprets as an integer - a number of units. That could also explain at least one memory leak. Would you be able to complete the trace and find who's passing a pointer in place of a value here?
I agree. The compiler should catch this, an evil cast, but it doesn't. I'm guessing this was an indirect function call through a table. Or a mis-cast void pointer.
I'm using the SAE J2735-2016 .ASN file. I'm trying to encode an RTCM message (decode works), but it's failing on the OCTET_STRING_uper.c (line 268). As you can see there's a really big number in this debug message which doesn't look right:
The C code is here: