Io Error: UnexpectedEof "failed to fill whole buffer" after writing many files

[bdemann]

I'm getting a strange error after attempting to open, read, and/or write many files. After about 20-25 writes I start getting the following error

Panicked at 'called `Result::unwrap()` on an `Err` value: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })', stable-fs-0.1.11/src/storage/types.rs:85:51

Even after doing a redeploy I am still unable to open any additional files after I hit what feels like some sort of limit.

Here is a simple canister that illustrates the problem.

After running dfx cansiter call canister write_25_files once it cannot be called again without this error popping up. Nor can read_25_file be called without this same error.

use std::{
    cell::RefCell,
    io::{self, Read, Write},
};

use ic_stable_structures::{
    memory_manager::{MemoryId, MemoryManager},
    DefaultMemoryImpl,
};

thread_local! {
    static MEMORY_MANAGER_REF_CELL: RefCell<MemoryManager<DefaultMemoryImpl>> = RefCell::new(MemoryManager::init(DefaultMemoryImpl::default()));
}

#[ic_cdk::init]
fn my_init() {
    let polyfill_memory =
        MEMORY_MANAGER_REF_CELL.with(|manager| manager.borrow().get(MemoryId::new(0)));
    ic_wasi_polyfill::init_with_memory(&[], &[], polyfill_memory);
}

#[ic_cdk::post_upgrade]
fn my_post_upgrade() {
    let polyfill_memory =
        MEMORY_MANAGER_REF_CELL.with(|thing| thing.borrow().get(MemoryId::new(0)));
    ic_wasi_polyfill::init_with_memory(&[], &[], polyfill_memory);
}

#[ic_cdk::update]
pub fn write_25_files() {
    let dir_name = "auto";
    let file_count: u8 = 25;
    std::fs::create_dir_all(dir_name).unwrap();
    for i in 0..file_count {
        let path = format!("{}/my_file_{}.txt", dir_name, i);
        ic_cdk::println!("Writing to {}", path);

        let mut file = std::fs::OpenOptions::new()
            .create(true)
            .write(true)
            .open(&path)
            .unwrap();
        let write_buff = [i; SIZE_OF_FILE];
        file.write(&write_buff).unwrap();
        drop(file);
    }
}

#[ic_cdk::update]
pub fn read_25_files() {
    let dir_name = "auto";
    let file_count: u8 = 25;
    std::fs::create_dir_all(dir_name).unwrap();
    for i in 0..file_count {
        let path = format!("{}/my_file_{}.txt", dir_name, i);
        ic_cdk::println!("Writing to {}", path);

        let mut file = std::fs::OpenOptions::new()
            .create(true)
            .read(true)
            .open(&path)
            .unwrap();
        let mut read_buf = [1, 1, 1];
        let num_bytes = file.read(&mut read_buf).unwrap();
        ic_cdk::println!("Read {} bytes {}", num_bytes, path);
    }
}

[wasm-forge]

I will have a look, but note, stable-fs v 0.1.11 has issues with the readdir implementation, it could cause errors when you work with multiple files. Please, try the latest version available from both the ic-wasi-polyfill (v0.3.17) and the stable-fs (v0.2.0).

[bdemann]

Thanks for the heads up! I tried the latest version (870e2cd0b533c03a45dbd3a516176a1fe261cea6) and still got the same error

The replica returned a replica error: reject code CanisterError, reject message Canister bkyz2-fmaaa-aaaaa-qaaaq-cai trapped explicitly: Panicked at 'called `Result::unwrap()` on an `Err` value: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })', stable-fs-0.2.0/src/storage/types.rs:85:51, error code None

[wasm-forge]

Hi Benjamin,

I've repeated running the same methods without errors (first writing 25 files, then reading 25 files, with and without upgrade). Which version of ic-stable-structures are you using? Apparently I am still missing some parts of your project (I assumed SIZE_OF_FILE is 300). Also, the error message is referencing src/storage/types.rs:85:51, the line does not exist in the original source code (does it?). Maybe try to check the ic-stable-structures is at least version 0.6.3 or send me more sources of your project.

[bdemann]

Oh woops, sorry I forgot to include SIZE_OF_FILE. I have it at 1_000_000. That should make all of the difference. With something as small as 300 I am also not ever running into the problem, at least not quickly.

I double checked everything and it looks like that was the only thing I missed.

Here is my Cargo.toml to be sure.

[package]
name = "globals_and_memory"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
crate-type = ["cdylib"]

[dependencies]
serde = '1.0.193'

candid = "0.10.5"
ic-cdk = "0.13.1"
ic-wasi-polyfill = { git = "https://github.com/wasm-forge/ic-wasi-polyfill", rev = "870e2cd0b533c03a45dbd3a516176a1fe261cea6", features = [
    "transient",
]}

ic-stable-structures = "0.6.3"

As for src/storage/types.rs:85:51 I was looking at that here https://github.com/wasm-forge/stable-fs/blob/4e3cb3b20b199d6ce23a38fc8304b8b07e7641f7/src/storage/types.rs#L85

[wasm-forge]

Hello Benjamin,

sorry for keeping you waiting. I have found the error and will provide a fix later today.

[wasm-forge]

Hello Benjamin,

I've published the error fix. Please check out the latest ic-wasi-polyfill version on crates.io. Unfortunately, the change is not backward compatible, and will break existing storage.

[bdemann]

I have tested this on version 0.5.0 of ic-was-polyfill and it is working as expected, thanks!