read_fst() checks for incomplete fst files

[j-kreis]

I am running R on a Mac with a M1 chip.

When I use the following code chunk R crashes with a 'floating point exception':

q = fst::fst("large_file.fst")
q = q[1:2000,]

Reading smaller chunks of the fst works fine.

[MarcusKlik]

Hi @ticarki, thanks for reporting!

It's hard to pinpoint the exact problem without the actual data. Does the following code lead to an identical error on your M1 chip?

dt <- data.frame(
    A = 1:1000,
    B = 1.1:10000.1
  )

path <- tempfile(".fst")
fst::write_fst(dt, path)

head(fst::fst(path)[1:2000, 1:2])
#>   A   B
#> 1 1 1.1
#> 2 2 2.1
#> 3 3 3.1
#> 4 4 4.1
#> 5 5 5.1
#> 6 6 6.1

also, what endianness is the M1 chip?

# on intel/AMD
.Platform$endian
#> [1] "little"

thanks

[j-kreis]

Hi @MarcusKlik, thanks for your answer! Blame on me, I probably used an incomplete fst file. Would it be possible to add a check for that? Not relevant anymore, but .Platform$endian returns "little"

[MarcusKlik]

Hi @ticarki, thanks for reporting back. Yes, adding a check for incomplete files would definitely add to the stability of fst. I will change the title of your issue and change it to a feature request!