ERROR [cubestore::remotefs::cleanup] error while getting created time for file - creation time is not available for the filesystem

[seatunneltrailuser]

Describe the bug While the Refresh Worker is performing pre-aggregations, we get multiple of this error

ERROR [cubestore::remotefs::cleanup] error while getting created time for file "919-sgnl1jjo.parquet":creation time is not available for the filesystem For every intermediary file created by the pre-aggregation process, CUBESTORE is also performing cleanup of these files. It is failing to purge these files due to this issue - Creation time is not available for the filesystem.

To Reproduce Steps to reproduce the behavior:

1. Run Docker containers as following Start the container for Router in VM1 Start the container for Worker1 in VM2 Start the container for Worker2 in VM3 Start the container for RefreshWorker in VM2 Start the container for API in VM3
2. The Router, Worker1, Worker2 is configured with the CUBESTORE_REMOTE_DIR to /cube/remote, and a shared network monted volume //data as /cube/remote for these containers
3. Pre-aggregations are automatically started, and after sometime we can start to see the the temporary files were not cleaned up due to this error.

Expected behavior These files are successfully cleaned up by the [cubestore::remotefs::cleanup] process.

Version: Docker image cube/cubestore:latest

Additional context Is it safe to clear these files manually ? Also, Are we missing any additional parameters to be set while starting up the RefreshWorker the job ?

[igorlukanin]

error while getting created time for file "919-sgnl1jjo.parquet":creation time is not available for the filesystem

@seatunneltrailuser What are you using as the filesystem? Could you please describe your setup?

[pasqualtroncone]

I have seen the same error on my tests. My host file system is xfs.

Here is my docker compose test setup

```yaml name: "cubejs" x-cubejs-image: &cubejs-image "docker.io/cubejs/cube:v0.35.22" x-cubestore-image: &cubestore-image "docker.io/cubejs/cubestore:v0.35.22" services: cubestore: image: *cubestore-image container_name: "cubejs-cubestore" hostname: "cubestore" networks: - "cubejs" environment: CUBESTORE_DATA_DIR: "/cube/data" CUBESTORE_LOG_LEVEL: "info" volumes: - "cubestore-data:/cube/data/" cube: image: *cubejs-image container_name: "cubejs" hostname: "cube" networks: - "cubejs" ports: - "4000:4000/tcp" - "5000:5000/tcp" environment: CUBEJS_CUBESTORE_HOST: "cubestore" CUBEJS_DB_HOST: "${CUBEJS_DB_HOST}" CUBEJS_DB_PORT: "${CUBEJS_DB_PORT}" CUBEJS_DB_NAME: "${CUBEJS_DB_NAME}" CUBEJS_DB_USER: "${CUBEJS_DB_USER}" CUBEJS_DB_PASS: "${CUBEJS_DB_PASS}" CUBEJS_DB_TYPE: "${CUBEJS_DB_TYPE}" CUBEJS_DB_CLICKHOUSE_READONLY: "true" CUBEJS_API_SECRET: "${CUBEJS_API_SECRET}" CUBEJS_PG_SQL_PORT: "${CUBEJS_PG_SQL_PORT}" CUBEJS_SQL_USER: "${CUBEJS_SQL_USER}" CUBEJS_SQL_PASSWORD: "${CUBEJS_SQL_PASSWORD}" CUBEJS_CACHE_AND_QUEUE_DRIVER: "cubestore" CUBEJS_LOG_LEVEL: "info" CUBEJS_DEV_MODE: "true" volumes: - "cube-data:/cube/conf/" - "./config/cubejs/cube.js:/cube/conf/cube.js:ro" - "./config/cubejs/model/:/cube/conf/model/custom/" depends_on: - "cubestore" volumes: cube-data: name: "cubejs-cube-data" cubestore-data: name: "cubejs-cubestore-data" networks: cubejs: name: "cubejs" internal: false ```

Log extracts

```log cubejs-cubestore | 2024-04-24T00:08:44.467Z INFO [cubestore::metastore::rocks_store] Persisting cachestore snapshot: done (2.25434ms) cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "1-hnwpuxtq.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "2-3nyem971.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "3-lujmbedj.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "4-xoct54kf.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "5-t2ipzm33.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "6-kujmssqi.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "8-i2idmyai.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "7-6hnq8uxm.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "9-c0fsdovn.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "10-ge6ds2ub.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "11-lcnahpyk.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "12-u7wwba87.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "14-fihpqpdt.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "13-zv3tgnxa.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "15-ncwvy49u.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "16-rfrsdovy.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "18-npj05axs.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "17-7ybmvqyi.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "19-xvwdfesw.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.967Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "20-8a0pftmc.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "22-cp3fonx5.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "21-swbvm5no.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "23-levsfozf.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "24-xu3nzyfw.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "26-emndusua.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "25-q9klsjyf.chunk.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "28-jvt5sxb2.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "29-oqc1wfzn.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "30-zv3g3peo.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "27-yia1mvib.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "31-wqi37b0k.parquet":creation time is not available for the filesystem cubejs-cubestore | 2024-04-24T00:08:58.968Z ERROR [cubestore::remotefs::cleanup] error while getting created time for file "32-8lhd05kd.parquet":creation time is not available for the filesystem ```

[thePermission]

what is the status of this? Will this be fixed by Cube? Is there a List of supported Filesystems?

[igorlukanin]

I've consulted with you-know-who and it looks like the root cause is the lack of the "creation time" support in XFS. I guess the workaround will be either to try -m crc=1 or switch to another filesystem:

@thePermission I guess any file system the stores a file "creation time" (e.g., ext4) should be OK. I'm going to add a note in the docs but it's outside of a roadmap to get this fixed. Contributions are always welcome, though!

[igorlukanin]

I would also add that using CUBESTORE_REMOTE_DIR is not recommended in production, so this whole issue should not be a huge blocker for production deployments: