solid-contrib / test-suite

An automated test of Solid specification technical compliance
MIT License
23 stars 10 forks source link

Avoid crash of overall table generation #77

Closed christophknabe closed 4 years ago

christophknabe commented 4 years ago

When generating the test suite summary table as documented in the README.md by the command egrep '[Tt]ests|earl:outcome' reports/* | docker run -i table-reporter the table generation crashes as follows:

Server                  LDP Basic               Websockets-pub-sub      RDF-fixtures        
gold                    0/90                    0/1                     2/45                
/report.js:101
    var perlBasedResult = `${table[serverName].perlBased.passedNumber}/${table[serverName].perlBased.totalNumber}`
                                                         ^
TypeError: Cannot read property 'passedNumber' of undefined
    at writeOutput (/report.js:101:58)
    at Socket.<anonymous> (/report.js:125:3)
    at Socket.emit (events.js:333:22)
    at endReadableNT (_stream_readable.js:1201:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

The reason are output lines from the rdf-fixtures tests containing the word "tests", as for example dc:title "- Delete resources that were set up in these tests"@en ;

I found out that matching the summary lines more exactly by egrep avoids this crash. See the following log:

~/test-suite$ egrep 'Tests:|tests run:|earl:outcome' reports/* | docker run -i table-reporter
Server                  LDP Basic               Websockets-pub-sub      RDF-fixtures        
gold                    0/90                    0/1                     2/45                
inrupt-pod-server       0/90                    0/1                     2/45                
node-solid-server       15/90                   0/1                     22/47               
trellis                 47/90                   0/1                     11/45               
wac-ldp                 56/90                   0/1                     8/47 

Please change the documentation, how to generate the test suite summary table in README.md to the following: egrep 'Tests:|tests run:|earl:outcome' reports/* | docker run -i table-reporter

kjetilk commented 4 years ago

Right, so, what we really should be doing here is to use an RDF parser to parse the EARL results, the current code is very preliminary. My main priority is currently to write more tests, to support the specification work, though.

christophknabe commented 4 years ago

I understand your priorities, but I only asked you to modify the README.md a bit, in order to avoid users of the test suite to run into the same problem as me. I could as well file a Pull Request with this one-line modification, if you prefer this.

kjetilk commented 4 years ago

Yes, it is actually very helpful, because if I need to step and out see what exactly I need to modify, I need to step out of the mental model on the thing I'm working on, and then it takes some time to get back in. I don't need to do that to just review and merge. So, PRs are always helpful.

And then, it is the bigger issue that using egrep and friends isn't robust, so my mind is closer to the long term solutions too :-)

christophknabe commented 4 years ago

By having pulled in the PR #79 this issue is solved.