search

ocramz / xeno

Fast Haskell XML parser
Other
120 stars 33 forks source link

stack bench on Travis goes haywire #20

Closed ocramz closed 6 years ago

ocramz commented 6 years ago 
Benchmark xeno-speed-bench: RUNNING...
benchmarking 4KB/hexml-dom
Progress 45/46: xeno-0.3.4time                 157.0 μs   (85.46 μs .. 227.1 μs)
                     0.510 R²   (0.339 R² .. 0.692 R²)
mean                 272.6 μs   (240.1 μs .. 316.5 μs)
std dev              52.40 μs   (31.67 μs .. 70.18 μs)
variance introduced by outliers: 92% (severely inflated)
Progress 45/46: xeno-0.3.4benchmarking 4KB/xeno-sax
Progress 45/46: xeno-0.3.4time                 14.16 μs   (8.660 μs .. 22.37 μs)
                     0.445 R²   (0.414 R² .. 0.995 R²)
mean                 59.53 μs   (59.53 μs .. 59.53 μs)
std dev              0.0 s      (0.0 s .. 0.0 s)
variance introduced by outliers: -9223372036854775808% (severely inflated)
Progress 45/46: xeno-0.3.4benchmarking 4KB/xeno-dom
Progress 45/46: xeno-0.3.4time                 173.4 μs   (23.20 μs .. 365.4 μs)
                     0.220 R²   (0.131 R² .. 0.996 R²)
xeno-speed-bench: ./Data/Vector/Generic.hs:245 ((!)): index out of bounds (-9223372036854775808,1000)
CallStack (from HasCallStack):
  error, called at ./Data/Vector/Internal/Check.hs:87:5 in vector-0.12.0.1-JlawpRjIcMJIYPJVsWriIA:Data.Vector.Internal.Check
Progress 45/46: xeno-0.3.4xeno-speed-bench: thread blocked indefinitely in an MVar operation
Benchmark xeno-speed-bench: ERROR
Completed 46 action(s).
ocramz commented 6 years ago

I think there's a bug for statistics as well, here

variance introduced by outliers: -9223372036854775808%
chrisdone commented 6 years ago

Is it 32-bit?

ocramz commented 6 years ago

Doesn't seem so, from the Travis build system info :

Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM
ocramz commented 6 years ago

what's strange is that the benchmarks for LTS-7.2 run fine on travis, albeit with strange numbers:

https://travis-ci.org/ocramz/xeno/jobs/379735031 

Benchmark xeno-memory-bench: RUNNING...
Case             Allocated  GCs
4kb/hexml/dom       -1,448    0
4kb/xeno/sax        -2,296    0
4kb/xeno/dom         7,656    0
31kb/hexml/dom      29,104    0
31kb/xeno/sax       -5,264    0
31kb/xeno/dom        6,056    0
211kb/hexml/dom    210,016    0
211kb/xeno/sax      21,696    0
211kb/xeno/dom   1,043,016    0
Benchmark xeno-memory-bench: FINISH
Benchmark xeno-speed-bench: RUNNING...
benchmarking 4KB/hexml-dom
time                 39.82 μs   (13.01 μs .. 94.81 μs)
                     0.210 R²   (0.178 R² .. 0.959 R²)
mean                 188.4 μs   (188.4 μs .. 188.4 μs)
std dev              0.0 s      (0.0 s .. 0.0 s)
variance introduced by outliers: -9223372036854775808% (severely inflated)
benchmarking 4KB/xeno-sax
time                 25.81 μs   (6.154 μs .. 72.70 μs)
                     0.166 R²   (0.148 R² .. 0.931 R²)
mean                 164.3 μs   (164.3 μs .. 164.3 μs)
std dev              0.0 s      (0.0 s .. 0.0 s)
variance introduced by outliers: -9223372036854775808% (severely inflated)
benchmarking 4KB/xeno-dom
time                 298.6 μs   (264.1 μs .. 330.9 μs)
                     0.848 R²   (0.734 R² .. 0.918 R²)
mean                 270.5 μs   (243.0 μs .. 314.7 μs)
std dev              87.23 μs   (46.31 μs .. 148.2 μs)
variance introduced by outliers: 97% (severely inflated)
benchmarking 4KB/hexpat-sax
time                 5.534 ms   (3.237 ms .. 8.412 ms)
ocramz commented 6 years ago

I've removed stack bench from the Travis script, for now