SQL Error: database disk image is malformed on directory creation

[Qiangong2]

The full error is:

SQL Error: database disk image is malformed
Trying to execute the following command :-

CREATE TABLE IF NOT EXISTS ESTIMATED_CTF_PARAMETERS(CTF_ESTIMATION_ID INTEGER PRIMARY KEY, CTF_ESTIMATION_JOB_ID INTEGER, DATETIME_OF_RUN INTEGER, IMAGE_ASSET_ID INTEGER, ESTIMATED_ON_MOVIE_FRAMES INTEGER, VOLTAGE REAL, SPHERICAL_ABERRATION REAL, PIXEL_SIZE REAL, AMPLITUDE_CONTRAST REAL, BOX_SIZE INTEGER, MIN_RESOLUTION REAL, MAX_RESOLUTION REAL, MIN_DEFOCUS REAL, MAX_DEFOCUS REAL, DEFOCUS_STEP REAL, RESTRAIN_ASTIGMATISM INTEGER, TOLERATED_ASTIGMATISM REAL, FIND_ADDITIONAL_PHASE_SHIFT INTEGER, MIN_PHASE_SHIFT REAL, MAX_PHASE_SHIFT REAL, PHASE_SHIFT_STEP REAL, DEFOCUS1 REAL, DEFOCUS2 REAL, DEFOCUS_ANGLE REAL, ADDITIONAL_PHASE_SHIFT REAL, SCORE REAL, DETECTED_RING_RESOLUTION REAL, DETECTED_ALIAS_RESOLUTION REAL, OUTPUT_DIAGNOSTIC_FILE TEXT, NUMBER_OF_FRAMES_AVERAGED INTEGER, LARGE_ASTIGMATISM_EXPECTED INTEGER, ICINESS REAL, TILT_ANGLE REAL, TILT_AXIS REAL );
From ../../src/core/database.cpp:29
int Database::ExecuteSQL(const char *)

cisTEM has been compiled using intel oneAPI 2023.1.0 (HPC, python, and base toolkits) on ubuntu 22.04. The GUI loads fine, but when you try to create a new project or open a previous one, you receive the error.

EDIT: This is the configuration I am using: ../configure CC=icc CXX=icpc CFLAGS="-O3 -ip -g -xCOMMON-AVX512 -restrict " CXXFLAGS="-O3 -ip -g -xCOMMON-AVX512 -restrict " --with-cuda=no

[bHimes]

Hi Qiangong

Both the Intel compiler version and the ubuntu version you are using are not tested by our team. My recommendation would be to compile using our developmental container (oneAPI 2021 and Ubuntu 20.0) and then run those binaries against a Singularity/Apptainer container. If you are interested in this, I can point you in the right direction.

Ben

[Qiangong2]

Hi Ben,

Thanks for the quick response. We're trying to unify our compiled applications as much as possible, so that includes moving to oneAPI 2023.1.0 and Ubuntu 22.04. Previously we've used sbgrid, but are trying to move away from that and focus on optimizing compiled applications.

It almost sounds like a permissions error, but no matter what user you run cisTEM as, you still get the malformed disk image error. Is there support for another database other than SQLite? Maybe MariaDB or Postgres?

-- Kurt

[jojoelfe]

Hi Kurt,

which filesystem are you trying to create the project on? I do remember there used to be issues with sqlite3 on certain filesystem (luster etc.).

No, sqlite is at the moment the only option.

Best,

Johannes

[Qiangong2]

Johannes,

It's an XFS filesystem. Probably doesn't matter, but it's an lvm volume group spread across three drives. No RAID configured either.

-- Kurt

[bHimes]

Hi Ben,

Thanks for the quick response. We're trying to unify our compiled applications as much as possible, so that includes moving to oneAPI 2023.1.0 and Ubuntu 22.04. Previously we've used sbgrid, but are trying to move away from that and focus on optimizing compiled applications.

It almost sounds like a permissions error, but no matter what user you run cisTEM as, you still get the malformed disk image error. Is there support for another database other than SQLite? Maybe MariaDB or Postgres?

-- Kurt

Hi Kurt - I fully appreciate the value in having a clear, unified, build environment. That is why I can't offer to troubleshoot builds beyond the build environment defined in our dev container. I hope you understand. Good luck optimizing your compiled applications! If you find a measurable improvement over supported versions, please let me know.

Ben

[bHimes]

@Qiangong2 to add a little more nuance: One reason this is (currently) out of scope is that Intel intentionally cripples AMD chips, and since the release of OneAPI the environmental variable hack no longer works.

We were able to find the function Intel calls internally to check chip type, precompile a dynamic lib with the same function name that always returns True, then use the LDPRELOAD hack so that this function is linked rather than the one Intel out in the MKL.

This only works for dynamic builds, and another solution was needed for the static builds we use for distribution.

All of this took considerable effort to track down and properly measure, and I see no reason Intel would not pull similar stunts in a new version, which makes upgrading a bit more involved than you might imagine.

( I have all AMD processors if that isn't obvious by now : )

Anyway, I was annoyed enough about this last go around, that I wrote a unit test to confirm that the Intel v AMD hack is working, so the only real barrier is making sure there are no weird issues with CUDA (even cuda 12.1 lists support only up to icc 2021.7)

I'm using both PTX (think assembly for gpus) and some funky approaches to pass maleable functions from host code to device using functors. These are other things that can produce very opaque or misleading behavior as a result of different optimizations or instruction sets chosen by the host compiler.

Anyway, I just wanted to give a little more context so you understand why we can't offer to support a change that may seem simple on the surface.

All that said, updating to Ubuntu 2022 is on my horizon, and I do have some profiling work planned, which would be the best time to compare a build with a new/unsupported icpc.

If you'd like, I can update this issue with my results. Otherwise, i'll close this one. LMK

Ben

[Qiangong2]

Thanks for the clarification. Updating the issue with your results would be great.

-- Kurt