X-Ryl669
commented
5 years ago Found from where the error message is emitted: com.sonar.sslr's lexer
Closed X-Ryl669 closed 5 years ago
X-Ryl669
commented
5 years ago Found from where the error message is emitted: com.sonar.sslr's lexer
guwirth
commented
5 years ago Hi @X-Ryl669,
thx for your feedback.
Please try to
Check also the encoding of your source files. Are they utf8?
Regards
X-Ryl669
commented
5 years ago Removing blanks does not make any difference, nor does removing backslashes. I'm using ASCII for encoding, but UTF-8 should work as well.
Also, removing the include directive allows the lexer to progress, but I'm getting syntax errors (obviously), thus no interesting report. So that's clearly something related to the include files.
X-Ryl669
commented
5 years ago After bisecting the include directories, I'm sure the parser & lexer is working (the source code appears in the web interface) but I'm still getting syntax errors (WARN: Source code parser: 30 syntax error(s) detected. Syntax errors could cause invalid software metric values. Root cause are typically missing includes, missing macros or compiler specific extensions.) yet there is nothing in the log saying from where the errors are coming from.
Is there a way to display the AST of each input file as it's being understood?
guwirth
commented
5 years ago @X-Ryl669 to get more information turn debug info on: https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Get-debug-information
If you wanna see the AST you can use the SSLR toolkit. Download and start it and copy the source code into it. You can also set includeDirectories there. https://github.com/SonarOpenCommunity/sonar-cxx/releases Play around with the code to narrow down the problem.
X-Ryl669
commented
5 years ago Debug is already on, yet it does not dump AST nor the reason for syntax error. I don't know how to use SSLR toolkit. I've the jar file but how am I supposed to use it ? Could not find any documentation except the sentence that's saying it's not a plugin.
guwirth
commented
5 years ago Could not find any documentation except the sentence that's saying it's not a plugin.
@X-Ryl669
Open Source File to open your sample or add code directly into Source Code windowParse Source Code to parse and create ASTConfiguration you can add include pathsX-Ryl669
commented
5 years ago Thanks! I've found some syntax that the SSLR parser does not like:
UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1>>> uartTask;This is accepted:
UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1> > > uartTask;Please notice that I'm compiling C++14 code, so it should be OK to use ">>>" to end a template declaration.
Also typeof and offsetof is not supported, so I've fixed a macro to return a cast pointer on the structure, but I guess it's expected ?
Yet, I'm able to parse 100% of the code (I've tried each file by hand) and it all parses correctly in SSLR toolkit, but sonar-scanner still report 2 syntax errors I can't find anywhere (even in the debug log).
X-Ryl669
commented
5 years ago Activated the rules tagged "tool-error" cxx:ParsingErrorRecovery allowed me to see where the parser failed:
It failed with such constructs:
extern const char cert_start[] asm("_binary_firmware_pho_by_crt_start");So to sum up the issues and workaround:
cxx:ParsingErrorRecovery rule.struct A { unsigned b:17; unsigned c:15; };), IMHO, this should be fixed tooguwirth
commented
5 years ago @X-Ryl669 thx for narrow this down.
typeof and offsetof problem to reproduce it.
ivangalkin
commented
5 years ago @guwirth
@X-Ryl669
includeDir IS a pain. Consider using json compile database if available. E.g. you might want to use https://github.com/rizsotto/Bear to intercept your compiler. X-Ryl669
commented
5 years ago I'm using xtensa's ESP32 compiler which is a port of gcc 5.2.0. containerof/offsetof are defined here
@ivangalkin I've already captured all include dir (not that hard to do, via a make -n + regexp). The issue is that if I use that list of directories, it fails. I have to bisect the list to find out the part that would allow the parser to work. IMHO, the sonar-scanner should report the complete SSLR's exception stack trace & message when such exception is generated, so at least, when bisecting we get a message telling where to look at instead of being blind. I'm sure it's possible to do it, since the SSLR toolkit is doing it already.
ivangalkin
commented
5 years ago @X-Ryl669
the sonar-scanner should report the complete SSLR's exception stack trace
I agree, that tracing of exception.toString() is often not enough. We should utilize more of ExceptionUtils.getStackTrace() and trace the context additionally. E.g. for the preprocessor it should be at least the current file (currentContextFile) and the stack of include files being processed (globalStateStack). That would help to identify, which include causes the problem.
guwirth
commented
5 years ago @X-Ryl669 I did some test with the SSLR toolkit 1.2.1:
Failing => new issue:
extern const char cert_start[] asm("_binary_firmware_pho_by_crt_start");Works, can't reproduce it:
UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1>>> uartTask;Works, can't reproduce it:
struct A { unsigned b:17; unsigned c:15; };Works, can't reproduce it:
struct S {
char c;
double d;
};
int main()
{
std::cout << "the first element is at offset " << offsetof(S, c) << '\n'
<< "the double is at offset " << offsetof(S, d) << '\n';
}C++/CLI not C++:
typeof / containerofDo you need my parsing errors ?
I think you need the (simplified) declaration of the failing code, so here it is:
#pragma pack(push, 1)
union ModeReg
{
uint32 reg;
struct
{
#ifdef BigEndian
uint32 meteringChannel : 1;
uint32 autoChannelSelect : 1;
uint32 channelAHighPassFilter : 1;
uint32 channelBHighPassFilter : 1;
uint32 voltageHighPassFilter : 1;
uint32 unused0 : 3;
uint32 energyMode : 2; //!< @sa EnergyMode
uint32 currentImbalanceJudgement : 2; //!< @sa CurrentImbalance
uint32 unused1 : 12;
uint32 registerID : 8; //!< Must be 0x14
#else
uint32 registerID : 8; //!< Must be 0x14
uint32 unused1 : 12;
uint32 currentImbalanceJudgement : 2; //!< @sa CurrentImbalance
uint32 energyMode : 2; //!< @sa EnergyMode
uint32 unused0 : 3;
uint32 voltageHighPassFilter : 1;
uint32 channelBHighPassFilter : 1;
uint32 channelAHighPassFilter : 1;
uint32 autoChannelSelect : 1;
uint32 meteringChannel : 1;
#endif
};
};
and
/** We are using CRTP here again to avoid virtual table cost.
You must implement a "void runTask()" method to be called within the task context */
template <typename T>
struct Task
{
inline static void implementTask(void * arg) { T * obj = static_cast<T*>(arg); obj->runTask(); }
/** Start the task that must implemet the "void runTask()" method
@param taskName The name of the task
@param stackSizeInWords The stack size in words (the actual size here is 4x larger than this number)
@param priority The priority of the task, from 0 for lowest priority to 32 highest
@param coreId The index of the core to pin the task to (0 or 1 on ESP32). If left by default, the task
is run on either core. */
inline void startTask(const char * taskName, int stackSizeInWords = 2048, int priority = 5, unsigned coreId = (unsigned)-1)
{
}
inline void stopTask()
{
}
inline bool started() const { return true; }
inline void delay(unsigned millisecond) { delayMs(millisecond); }
Task() {}
};
/** The UART base code without event queue in interrupt */
template <int uartNum, int baudRate, int dataBitCount, char parity, int stopBitCount>
struct UART
{
enum { Port = uartNum, WithQueue = 0 };
UART()
{
}
int read(uint8 * data, uint32 len, const int timeoutMs = 100)
{
}
int write(const uint8 * data, const uint32 len)
{
}
};
/** The UART task that's dealing with UART events */
template<typename Controller>
struct UARTTask : public Task< UARTTask<Controller> >, public Controller
{
Timer timer;
void runTask()
{
uint32 counter = 0;
static_cast<Controller*>(this)->started(timer);
while(true)
{
if (timer.waitForTimer(200))
{
static_cast<Controller*>(this)->quiescent(counter++);
}
}
}
void start() { this->startTask("uart", 2560, 12); }
UARTTask() { }
~UARTTask() { this->stopTask(); }
};
template <typename UART>
struct BL6523GXController : public UART
{
BL6523GXController() {}
// The main task is started
void started(Timer & timer)
{
// Then we can trigger the timer
timer.triggerPeriodically(400000); // That's 1 / 2.5Hz in us
}
void quiescent(uint32 counter)
{
}
};
UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1>>> uartTask;and
// Found in compiler's include library
#define esp_container_of( ptr, type, member ) ( { \
const typeof( ( (type * )0 )->member ) *__mptr = ( ptr ); \
( type * )( ( char * )__mptr - offsetof( type,member ) );} )
// Used in:
static CgiStatus logRequest(HttpdConnData *connData)
{
char * ip = esp_container_of(connData, RtosConnType, connData)->ip;
Log(Logger::Info|Logger::Network, "%s (%s)%s [%d.%d.%d.%d]", HttpMethodStr[connData->requestType], connData->hostName, connData->url, ip[0], ip[1], ip[2], ip[3]);
return HTTPD_CGI_NOTFOUND;
}
I'll give the exact parsing error when I'm back at home.
X-Ryl669
commented
5 years ago Here is the exception from the CxxToolkit for container_of:
(sorry CxxToolkit does not allow copy & paste)
Here is the exception for the >>> template:
The source code is like this:
httpdSend(connData, "}", 1);
return HTTPD_CGI_DONE;
}
extern UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1>>> uartTask;
static CgiStatus devUART(HttpdConnData *connData)
{
uint32 addr, value, write;
if (!devCommon(connData, addr, value, write)) return HTTPD_CGI_DONE;
if (!addr)
{@X-Ryl669 samples above are working with SSLR-1.2.1, except:
#define esp_container_of( ptr, type, member ) ( { \
const typeof( ( (type * )0 )->member ) *__mptr = ( ptr ); \
( type * )( ( char * )__mptr - offsetof( type,member ) );} )
static CgiStatus logRequest(HttpdConnData *connData)
{
char * ip = esp_container_of(connData, RtosConnType, connData)->ip;
}Problem is this esp_container_of macro. A quick test with VS also failed?
X-Ryl669
commented
5 years ago What's VS ?
X-Ryl669
commented
5 years ago Concerning the bitfield issue: The parser seems to accept them, but they are not understood as bitfield. In fact, this leads to strange errors with the rules about bad type (I'll copy them when I'm back home again). This looks like a parser output error (it does not produce a good AST), even if it continues parsing the rest of the code correctly.
The esp_container_of macro compiles without any issue with my compiler. I think the parser chokes on the 2 statements the macro expands into ({ type a = something; (rettype*)(a - 35); }) (please notice the 2 ;) since it's used in a variable declaration statement. I'm not sure it's allowed in the C++ standard, but since it's not my code, I'm not able to modify it. I can live with it since I can redefine the macro to nothing easily.
guwirth
commented
5 years ago @X-Ryl669 VS = VisualStudio
Seems to be no C++?
char * ip = esp_container_of(connData, RtosConnType, connData)->ip;
char * ip = ({ type a = something; (rettype*)(a - 35); })->ip;I don't know for sure. When you have multiple statement per line (like i+=printf("something"),5), it's like if you had printf("something"); i+= 5 in that example).
Here we are in a variable declaration statement, so the parenthesis is mandatory, else it would be interpreted as a second declaration: int i = printf(), 5 is wrong, but int i = (printf(),5) is correct.
So the enclosing parenthesis is mandatory.
I don't know if the ; is considered like a comma here, I don't think so.
Since the part is enclosed in {}, it should be considered as an expression, and the expression's result is the last statement in it, so it should be equivalent to char * ip = (rettype*)(a - 35)
In the end, the expression resumes to int a = ({first statement; second statement;}) leading to a being set to the evaluation of the second statement.
This is accepted by all C++ compilers, so I would say it's correct (see this ideone ). I agree that's not usual code, but if it's compliant, it should be parsed.
X-Ryl669
commented
5 years ago Concerning the UARTTask error, it looks like a preprocessor issue (in the screenshot above, the "error" line does not contain extern UARTTask<BL6523GXController<UART<UART_NUM_2, 4800, 8, 'N', 1>>> uartTask; but only extern UARTTask>> uartTask; which is clearly an invalid statement).
So the error is not the number of > here like I initially thought (and that would explain why a simple test works for you) but something wrong with the preprocessor.
This is strange since there is no macro involved in this statement (UART_NUM_2 is an enum's member and I've given the source declaration for all other template's struct).
I wonder if there is some macro declaration that's interfering in some included file with the parser and not while compiling my code with a real compiler.
Is there a way to capture the preprocessed string of the SSLR toolkit ?
guwirth
commented
5 years ago @X-Ryl669 which compiler are you using?
X-Ryl669
commented
5 years ago It's a Xtensa ESP32 cross compiler based on GCC5.2.
guwirth
commented
5 years ago see #1685
I've the same issue as #872 but it was closed without resolving. I'm getting a
Lexer error: Unable to lex urlfor each source file.Here's the output of the sonar-scanner's log:
I've added the macros for the compiler extensions, and here's my
project.properties: