search

fstpackage / fst

Lightning Fast Serialization of Data Frames for R
http://www.fstpackage.org/fst/
GNU Affero General Public License v3.0
614 stars 42 forks source link

Support big-endian platforms #248

Open dipterix opened 3 years ago

dipterix commented 3 years ago

Hi @MarcusKlik

I recently submitted a package to CRAN, it gets this error during check with Solaris (https://cran.r-project.org/web/checks/check_results_raveio.html)

     *** caught segfault ***
    address e25a35d4, cause 'memory not mapped'

This error comes from fst (https://cran.r-project.org/web/checks/check_results_fst.html)

Version: 0.9.2
Check: tests
Result: ERROR
     Running ‘testthat.R’ [62s/100s]
    Running the tests in ‘tests/testthat.R’ failed.
    Complete output:
     >
     > # required packages
     > library(testthat)
     > library(fst)
     > library(data.table)
     > library(lintr)
     > library(bit64)
     Loading required package: bit

     Attaching package: 'bit'

     The following object is masked from 'package:data.table':

     setattr

     The following object is masked from 'package:base':

     xor

     Attaching package bit64
     package:bit64 (c) 2011-2017 Jens Oehlschlaegel
     creators: integer64 runif64 seq :
     coercion: as.integer64 as.vector as.logical as.integer as.double as.character as.bitstring
     logical operator: ! & | xor != == < <= >= >
     arithmetic operator: + - * / %/% %% ^
     math: sign abs sqrt log log2 log10
     math: floor ceiling trunc round
     querying: is.integer64 is.vector [is.atomic} [length] format print str
     values: is.na is.nan is.finite is.infinite
     aggregation: any all min max range sum prod
     cumulation: diff cummin cummax cumsum cumprod
     access: length<- [ [<- [[ [[<-
     combine: c rep cbind rbind as.data.frame
     WARNING don't use as subscripts
     WARNING semantics differ from integer
     for more help type ?bit64

     Attaching package: 'bit64'

     The following objects are masked from 'package:base':

     %in%, :, is.double, match, order, rank

     > library(nanotime)
     >
     > # some helper functions
     > source("testthat/helper_fstwrite.R") # cross-version testing
     >
     > # run tests
     > test_check("fst")
     ── 1. Failure: Medium factor with a single NA level (@test_factor.R#130) ──────
     `dt` not equal to `data`.
     Component "A": Mean relative difference: 0.6666124

     ── 2. Failure: Medium factor with a single NA level (@test_factor.R#130) ──────
     to_frame(dt[offset:nrow(dt), , drop = FALSE]) not equal to `data`.
     Component "A": Mean relative difference: 0.6666124

     ── 3. Failure: Medium factor with a single NA level (@test_factor.R#130) ──────
     to_frame(dt[1:(nrow(dt) - cap), , drop = FALSE]) not equal to `data`.
     Component "A": Mean relative difference: 0.6666124

     ── 4. Failure: Medium factor with a single NA level (@test_factor.R#130) ──────
     to_frame(dt[offset:(nrow(dt) - cap), , drop = FALSE]) not equal to `data`.
     Component "A": Mean relative difference: 0.6666124

     ── 5. Failure: Single uncompressed vectors (@test_fst.R#107) ──────────────────
     `sub_dt` not equal to `data`.
     Component "Xint": Mean relative difference: 0.6666124
     args: col: Xint | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 1,2,3,4,5,6,7,8,9,10 head data: 1,2,3,4,5,6,7,8,9,10 unequals: 4096 uneq rows sub_dt1 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 8193,8194,8195,8196,8197,8198,8199,8200,8201,8202,8203,8204,8205,8206,8207,8208,8209,8210,8211,8212,8213,8214,8215,8216,8217

     ── 6. Failure: Single uncompressed vectors (@test_fst.R#107) ──────────────────
     `sub_dt` not equal to `data`.
     Component "Ylog": 'is.NA' value mismatch: 3473 in current 3418 in target
     args: col: Ylog | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: TRUE,FALSE,NA,TRUE,FALSE,FALSE,TRUE,TRUE,TRUE,NA head data: TRUE,FALSE,NA,TRUE,FALSE,FALSE,TRUE,TRUE,TRUE,NA unequals: NA uneq rows sub_dt1 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA uneq rows sub_dt2 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA uneq rows sub_dt3 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA

     ── 7. Failure: Single uncompressed vectors (@test_fst.R#107) ──────────────────
     `sub_dt` not equal to `data`.
     Component "Zdoub": Mean relative difference: 1.389547
     args: col: Zdoub | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 0.892903014538854,-0.0460623216955318,-0.6093979383735,0.8708187551391,2.90528384287998,1.58217477578285,-1.35592735645172,1.68711472552246,-1.31324910050173,-0.340544715103315 head data: 0.892903014538854,-0.0460623216955318,-0.6093979383735,0.8708187551391,2.90528384287998,1.58217477578285,-1.35592735645172,1.68711472552246,-1.31324910050173,-0.340544715103315 unequals: 4096 uneq rows sub_dt1 1.65354450901977,2.04484180750652,-0.79672592607758,1.30520546040914,0.210101164046691,-1.02482874855373,0.0630000162992012,0.365654390401275,1.95218345428785,0.009445639613312,-1.5880683431947,1.75335242758132,-0.369319879987769,-0.607217609659549,0.315236168361736,0.147413338698303,-1.74131102367748,0.986234772369856,-0.273644178863937,-2.10167553883039,0.0914431228442858,-1.72105871133585,-0.321389857770328,-1.62671636162376,0.979319378296916 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 -2.39119428479779,-2.51934031980168,-1.59693973392773,0.246635324191922,1.13914473577017,1.93981412520328,1.01300755932906,0.90004442241108,0.477509166299311,0.0769122555483165,-0.015060677925639,-0.211661312308505,-0.8524093360692,0.900504932670958,-1.0275083226766,-0.446179314021679,-0.463365303130139,0.328669988149129,-1.50708224880752,1.35809990455963,0.0315688638693045,-1.02249636455192,-0.557337471849977,-0.757639800649845,-0.981985382707096

     ── 8. Failure: Single uncompressed vectors (@test_fst.R#107) ──────────────────
     `sub_dt` not equal to `data`.
     Component "WFact": 3578 string mismatches
     args: col: WFact | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: ZAK,KBM,EIS,ZIM,ZAK,ZAK,ZAK,XWU,KBM,EIS head data: ZAK,KBM,EIS,ZIM,ZAK,ZAK,ZAK,XWU,KBM,EIS unequals: 3578 uneq rows sub_dt1 NK,XWU,ZIM,LS,PHTX,PHTX,XWU,EIS,XWU,NK,ZAK,XWU,LS,NK,ZAK,KBM,PHTX,KBM,XWU,EIS,ZAK,LS,NK,ZIM,ZIM uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4115,4116,4117,4118,4119,4120,4122,4123 uneq rows sub_dt3 ZAK,KBM,EIS,ZIM,ZAK,ZAK,ZAK,XWU,KBM,EIS,KBM,ZAK,EIS,EIS,KBM,ZIM,EIS,ZIM,ZAK,KBM,NK,ZIM,LS,PHTX,KBM

     ── 9. Failure: Single uncompressed vectors (@test_fst.R#107) ──────────────────
     `sub_dt` not equal to `data`.
     Component "Ordered": 3583 string mismatches
     args: col: Ordered | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: WF,A,PGDE,VT,VT,PGDE,E,N,A,QH head data: WF,A,PGDE,VT,VT,PGDE,E,N,A,QH unequals: 3583 uneq rows sub_dt1 E,QH,A,E,QH,QH,A,QH,VT,E,E,VT,E,BRE,E,N,E,N,E,N,WF,BRE,BRE,E,BRE uneq rows sub_dt2 4097,4098,4099,4100,4101,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4119,4120,4121,4122,4123 uneq rows sub_dt3 WF,A,PGDE,VT,VT,E,N,A,QH,A,QH,E,A,A,N,WF,A,E,QH,PGDE,BRE,VT,E,VT,QH

     ── 10. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "Date": Mean relative difference: 0.6794327
     args: col: Date | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 1981-09-05,1985-12-15,1985-02-09,1984-06-22,1979-04-03,1978-02-24,1984-04-30,1982-01-27,1988-08-11,1970-11-12 head data: 1981-09-05,1985-12-15,1985-02-09,1984-06-22,1979-04-03,1978-02-24,1984-04-30,1982-01-27,1988-08-11,1970-11-12 unequals: 4095 uneq rows sub_dt1 1985-04-17,1983-09-17,1977-04-11,1980-07-01,1992-01-30,1976-09-26,1989-08-01,1976-11-13,1984-09-25,1974-11-15,1980-10-19,1992-09-29,1996-01-30,1980-01-04,1971-11-10,1980-11-05,1970-11-12,1976-06-15,1991-05-23,1978-07-08,1974-09-19,1972-04-07,1995-08-15,1985-03-31,1989-04-09 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 1993-11-20,1992-06-23,1980-06-05,1976-09-25,1985-05-20,1971-09-03,1974-01-22,1979-04-13,1986-07-24,1972-02-04,1975-09-05,1973-04-26,1971-12-28,1980-03-01,1983-09-07,1996-12-03,1972-07-13,1982-01-17,1994-06-16,1984-12-09,1978-03-27,1991-02-19,1984-01-09,1976-03-15,1996-11-30

     ── 11. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "DateDouble": Mean relative difference: 0.1813955
     args: col: DateDouble | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 2015-01-02,2015-01-03,2015-01-04,2015-01-05,2015-01-06,2015-01-07,2015-01-08,2015-01-09,2015-01-10,2015-01-11 head data: 2015-01-02,2015-01-03,2015-01-04,2015-01-05,2015-01-06,2015-01-07,2015-01-08,2015-01-09,2015-01-10,2015-01-11 unequals: 4096 uneq rows sub_dt1 2026-03-21,2026-03-22,2026-03-23,2026-03-24,2026-03-25,2026-03-26,2026-03-27,2026-03-28,2026-03-29,2026-03-30,2026-03-31,2026-04-01,2026-04-02,2026-04-03,2026-04-04,2026-04-05,2026-04-06,2026-04-07,2026-04-08,2026-04-09,2026-04-10,2026-04-11,2026-04-12,2026-04-13,2026-04-14 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 2037-06-07,2037-06-08,2037-06-09,2037-06-10,2037-06-11,2037-06-12,2037-06-13,2037-06-14,2037-06-15,2037-06-16,2037-06-17,2037-06-18,2037-06-19,2037-06-20,2037-06-21,2037-06-22,2037-06-23,2037-06-24,2037-06-25,2037-06-26,2037-06-27,2037-06-28,2037-06-29,2037-06-30,2037-07-01

     ── 12. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "Integer64": Mean relative difference: 2.032993
     args: col: Integer64 | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 2345612345679,2345612345679,8714567890,8714567890,8714567890,10,10,8714567890,8714567890,10 head data: 2345612345679,2345612345679,8714567890,8714567890,8714567890,10,10,8714567890,8714567890,10 unequals: 2704 uneq rows sub_dt1 8714567890,8714567890,10,2345612345679,2345612345679,10,8714567890,2345612345679,10,2345612345679,10,10,2345612345679,10,2345612345679,10,10,10,2345612345679,10,8714567890,10,2345612345679,10,8714567890 uneq rows sub_dt2 4098,4099,4100,4101,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4114,4118,4119,4120,4121,4122,4124,4125,4126,4129,4131 uneq rows sub_dt3 2345612345679,10,2345612345679,10,10,8714567890,10,10,2345612345679,8714567890,8714567890,2345612345679,10,2345612345679,10,8714567890,8714567890,2345612345679,8714567890,2345612345679,10,2345612345679,10,2345612345679,10

     ── 13. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "Nanotime": Mean relative difference: 0.2238657
     args: col: Nanotime | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 1843155,1587545,1606103,1291552,1752254,1660217,1762777,1903074,1205443,1272100 head data: 1843155,1587545,1606103,1291552,1752254,1660217,1762777,1903074,1205443,1272100 unequals: 4096 uneq rows sub_dt1 1035443,1888338,1909472,1036702,1433383,1554381,1194767,1980198,1658299,1208527,1191313,1484528,1177063,1037892,1146552,1338340,1441528,1943208,1814953,1496786,1159298,1998079,1940621,1788368,1142421 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 1938828,1073580,1617608,1403889,1749021,1721391,1498508,1992423,1994215,1379800,1778154,1560801,1093607,1490249,1450448,1332995,1740140,1888848,1544333,1618549,1650036,1400300,1927937,1792877,1472937

     ── 14. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "Difftime": Mean relative difference: 0.6666125
     args: col: Difftime | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 0.998980522155762,1.99898052215576,2.99898052215576,3.99898052215576,4.99898052215576,5.99898052215576,6.99898052215576,7.99898052215576,8.99898052215576,9.99898052215576 head data: 0.998980522155762,1.99898052215576,2.99898052215576,3.99898052215576,4.99898052215576,5.99898052215576,6.99898052215576,7.99898052215576,8.99898052215576,9.99898052215576 unequals: 4096 uneq rows sub_dt1 4096.99898052216,4097.99898052216,4098.99898052216,4099.99898052216,4100.99898052216,4101.99898052216,4102.99898052216,4103.99898052216,4104.99898052216,4105.99898052216,4106.99898052216,4107.99898052216,4108.99898052216,4109.99898052216,4110.99898052216,4111.99898052216,4112.99898052216,4113.99898052216,4114.99898052216,4115.99898052216,4116.99898052216,4117.99898052216,4118.99898052216,4119.99898052216,4120.99898052216 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 8192.99898052216,8193.99898052216,8194.99898052216,8195.99898052216,8196.99898052216,8197.99898052216,8198.99898052216,8199.99898052216,8200.99898052216,8201.99898052216,8202.99898052216,8203.99898052216,8204.99898052216,8205.99898052216,8206.99898052216,8207.99898052216,8208.99898052216,8209.99898052216,8210.99898052216,8211.99898052216,8212.99898052216,8213.99898052216,8214.99898052216,8215.99898052216,8216.99898052216

     ── 15. Failure: Single uncompressed vectors (@test_fst.R#107) ─────────────────
     `sub_dt` not equal to `data`.
     Component "DiffTime_int": Mean relative difference: 0.6667209
     args: col: DiffTime_int | from: 1 | to: 10000 | setColumns: | compress: 0 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 0,1,2,3,4,5,6,7,8,9 head data: 0,1,2,3,4,5,6,7,8,9 unequals: 4096 uneq rows sub_dt1 4096,4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 8192,8193,8194,8195,8196,8197,8198,8199,8200,8201,8202,8203,8204,8205,8206,8207,8208,8209,8210,8211,8212,8213,8214,8215,8216

     ── 16. Failure: Single weakly compressed vectors (@test_fst.R#122) ────────────
     `sub_dt` not equal to `data`.
     Component "Xint": 'is.NA' value mismatch: 1 in current 0 in target
     args: col: Xint | from: 1 | to: 10000 | setColumns: | compress: 30 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: 1,2,3,4,5,6,7,8,9,10 head data: 1,2,3,4,5,6,7,8,9,10 unequals: NA uneq rows sub_dt1 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt2 4097,4098,4099,4100,4101,4102,4103,4104,4105,4106,4107,4108,4109,4110,4111,4112,4113,4114,4115,4116,4117,4118,4119,4120,4121 uneq rows sub_dt3 4,4,65781,65782,0,12189696,0,0,117440516,65540,-16711433,9502968,-323289088,0,0,603979776,65540,0,9568504,0,0,0,0,0,458759

     ── 17. Failure: Single weakly compressed vectors (@test_fst.R#122) ────────────
     `sub_dt` not equal to `data`.
     Component "Ylog": 'is.NA' value mismatch: 3052 in current 3418 in target
     args: col: Ylog | from: 1 | to: 10000 | setColumns: | compress: 30 | tot_length 10000 cols sub_dt: 1 , rows sub_dt: 10000 cols data: 1 , rows data: 10000 head sub_dt: TRUE,FALSE,NA,TRUE,FALSE,FALSE,TRUE,TRUE,TRUE,NA head data: TRUE,FALSE,NA,TRUE,FALSE,FALSE,TRUE,TRUE,TRUE,NA unequals: NA uneq rows sub_dt1 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA uneq rows sub_dt2 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA uneq rows sub_dt3 NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA

     *** caught segfault ***
     address d83, cause 'memory not mapped'

     Traceback:
     1: fstretrieve(file_name, columns, from, to)
     2: read_fst(path, columns, from, to, as_data_table)
     3: fstreadproxy("FactorStore/data1.fst", columns = sel_columns, from = from, to = to)
     4: test_write_read(x, compress = 30L)
     5: FUN(X[[i]], ...)
     6: lapply(X = X, FUN = FUN, ...)
     7: sapply(col_names, function(x) { test_write_read(x, compress = 30L)})
     8: eval(code, test_env)
     9: eval(code, test_env)
     10: withCallingHandlers({ eval(code, test_env) if (!handled && !is.null(test)) { skip_empty() }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning, message = handle_message, error = handle_error)
     11: doTryCatch(return(expr), name, parentenv, handler)
     12: tryCatchOne(expr, names, parentenv, handlers[[1L]])
     13: tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
     14: doTryCatch(return(expr), name, parentenv, handler)
     15: tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), names[nh], parentenv, handlers[[nh]])
     16: tryCatchList(expr, classes, parentenv, handlers)
     17: tryCatch(withCallingHandlers({ eval(code, test_env) if (!handled && !is.null(test)) { skip_empty() }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning, message = handle_message, error = handle_error), error = handle_fatal, skip = function(e) { })
     18: test_code(desc, code, env = parent.frame())
     19: test_that("Single weakly compressed vectors", { sapply(col_names, function(x) { test_write_read(x, compress = 30L) })})
     20: eval(code, test_env)
     21: eval(code, test_env)
     22: withCallingHandlers({ eval(code, test_env) if (!handled && !is.null(test)) { skip_empty() }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning, message = handle_message, error = handle_error)
     23: doTryCatch(return(expr), name, parentenv, handler)
     24: tryCatchOne(expr, names, parentenv, handlers[[1L]])
     25: tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
     26: doTryCatch(return(expr), name, parentenv, handler)
     27: tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), names[nh], parentenv, handlers[[nh]])
     28: tryCatchList(expr, classes, parentenv, handlers)
     29: tryCatch(withCallingHandlers({ eval(code, test_env) if (!handled && !is.null(test)) { skip_empty() }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning, message = handle_message, error = handle_error), error = handle_fatal, skip = function(e) { })
     30: test_code(NULL, exprs, env)
     31: source_file(path, new.env(parent = env), chdir = TRUE, wrap = wrap)
     32: force(code)
     33: doWithOneRestart(return(expr), restart)
     34: withOneRestart(expr, restarts[[1L]])
     35: withRestarts(testthat_abort_reporter = function() NULL, force(code))
     36: with_reporter(reporter = reporter, start_end_reporter = start_end_reporter, { reporter$start_file(basename(path)) lister$start_file(basename(path)) source_file(path, new.env(parent = env), chdir = TRUE, wrap = wrap) reporter$.end_context() reporter$end_file() })
     37: FUN(X[[i]], ...)
     38: lapply(paths, test_file, env = env, reporter = current_reporter, start_end_reporter = FALSE, load_helpers = FALSE, wrap = wrap)
     39: force(code)
     40: doWithOneRestart(return(expr), restart)
     41: withOneRestart(expr, restarts[[1L]])
     42: withRestarts(testthat_abort_reporter = function() NULL, force(code))
     43: with_reporter(reporter = current_reporter, results <- lapply(paths, test_file, env = env, reporter = current_reporter, start_end_reporter = FALSE, load_helpers = FALSE, wrap = wrap))
     44: test_files(paths, reporter = reporter, env = env, stop_on_failure = stop_on_failure, stop_on_warning = stop_on_warning, wrap = wrap)
     45: test_dir(path = test_path, reporter = reporter, env = env, filter = filter, ..., stop_on_failure = stop_on_failure, stop_on_warning = stop_on_warning, wrap = wrap)
     46: test_package_dir(package = package, test_path = test_path, filter = filter, reporter = reporter, ..., stop_on_failure = stop_on_failure, stop_on_warning = stop_on_warning, wrap = wrap)
     47: test_check("fst")
     An irrecoverable exception occurred. R is aborting now ...
Flavor: r-patched-solaris-x86

Would you like to investigate?

MarcusKlik commented 3 years ago

Hi @dipterix, thanks for your question (and congratulations on your raveio CRAN submission 👍)

The Solaris errors are due to fst being build for little-endian machines and Solaris is big-endian. You can indicate this by setting

SystemRequirements: little-endian platform

in file DESCRIPTION in the source of your package. That way, CRAN will still check your package on a Solaris machine but not fail your submission because of these errors :-)

dipterix commented 3 years ago

Uh... I thought fst once supported big-endian system, or Solaris is unsupported all the time? Also do you have any plan to enable big-endian support because neither LZ4 nor ZSTD forces endianess?

MarcusKlik commented 3 years ago

Ha @dipterix, it's possible to use fst on a big-endian system without problems if you just write and read on the same system. But when a fst file written on a little-endian system is read on a big-endian system (or vice versa), that's when problems start.

In earlier submissions, CRAN checks were successful because the unit tests did not include reading in default fst files created on a little-endian system...

Making fst endianness-proof is certainly on the feature list. The main issue is adapting the custom byte-shift filters that fst uses to increase (de-) compression speeds. Also, the bytes in the meta-data blocks need to be reversed for big-endian systems...

thanks for reporting!

dipterix commented 3 years ago

I see. Is it possible to store pre-designed bytes within the meta block that indicates the endianess, for example one or multiple 00001111 (or something that has very low probability to occur) at both ends of the file. In this case, if the bytes read are 11110000, then you know the endian should be changed, otherwise if the bytes are not 00001111, just assume there is no overhead. I guess you must have clever way to handle this without breaking compatibilities : D

MarcusKlik commented 3 years ago

Hi @dipterix, you're exactly right :-)

In fact, I have already prepared a binary flag in the header of the file that stores the endianess, see this specification. But at the moment, the value is not used to check against the systems endianess, because DESCRIPTION contains SystemRequirements: little-endian platform anyway...