SQList Corruption Issues

[ancker010]

I keep running into SQLite corruption issues with my instance. (Docker) There doesn't seem to be any pattern I can ascertain. It'll happen anywhere from 2-3 days after starting to several weeks. The only fix is to stop the container, delete the SQLite files, and restart.

Disk is not full, no other containers on the same system exhibit any issues.

2023-08-03 14:53:18 ERROR evebox::eve::processor: Failed to commit events (will try again): sqlite error: database disk image is malformed 2023-08-03 14:53:19 ERROR evebox::sqlite::importer: Failed to commit events: error=database disk image is malformed, source=Some(Error { code: DatabaseCorrupt, extended_code: 267 }) 2023-08-03 14:53:19 ERROR evebox::eve::processor: Failed to commit events (will try again): sqlite error: database disk image is malformed 2023-08-03 14:53:19 ERROR evebox::sqlite::retention: Database retention job failed: database disk image is malformed 2023-08-03 14:53:20 ERROR evebox::sqlite::importer: Failed to commit events: error=database disk image is malformed, source=Some(Error { code: DatabaseCorrupt, extended_code: 267 }) 2023-08-03 14:53:20 ERROR evebox::eve::processor: Failed to commit events (will try again): sqlite error: database disk image is malformed

[jasonish]

Can't say I've ever seen this. I do have one SQLite instance where the DB is set to 100GB for max size, so there is a lot of thrashing on it as events constantly get purged as they are added. I'm using Docker for this as well.

What is the underlying file system? What about the Docker driver for the volume? I'm just using default bind or host mounts.

[ancker010]

Underlying filesystem is ext4 on an SSD. I do directory mapping instead of creating a docker volume. So just...

  volumes:
      - /storage/suricata/logs:/var/log/suricata
      - /storage/suricata/config:/etc/suricata
      - /storage/suricata/lib:/var/lib/suricata

I have my settings at 7 days and 20GB should I consider bumping that up? I don't care to keep the events super long, so I set them pretty low.

[jasonish]

Ok, I'm on ext4 as well, ssd. One different between your setup and mine is that I also have my EveBox data directory on a mapped volume, this could be the difference. My docker-compose.yml entry for the SQLite test instance:

  evebox-sqlite-latest:
    image: docker.io/jasonish/evebox:latest
    restart: unless-stopped
    ports:
      - "3002:5636"
    command: -vvv -c /evebox.yaml -D /data --datastore sqlite
    volumes:
      - ./evebox.yaml:/evebox.yaml:z
      - ./app-data/sqlite:/data:z

[ancker010]

Ugh, disregard some the above pastes, I accidentally grabbed the suricata mounts. This is my evebox mounts.

    volumes:
      - /storage/suricata/evebox:/config
      - /storage/suricata/logs:/var/log/suricata

command: -D /config --datastore sqlite --input /var/log/suricata/eve.json

[jasonish]

Ok, so your setup is very close to mine, so I'm not actually sure at the moment. I only have a file size limit on my database at the moment, I'll add a time limit like you do as well.

[jasonish]

Closing as stale and was not able to replicate.