PostgreSQL does not work on GlusterFS (but works on kNFS)

warmanton commented 3 years ago

Description of problem: Trying to get persistent storage for docker swam. Testing on single node swarm + two nodes with Gluster server. Trying to deploy gitlab instance with thee bind mounts targeted on GlusterFS mount point. Volumes look like thes:

volumes:
  - /mnt/g/gitlab/data:/var/opt/gitlab
  - /mnt/g/gitlab/logs:/var/log/gitlab
  - /mnt/g/gitlab/config:/etc/gitlab

Got failed docker container with messages like these:

postgresql_extension[btree_gist] action enable
- postgresql_query[enable btree_gist extension] action run
  - execute[enable btree_gist extension (postgresql)] action run [execute] ERROR: unexpected data beyond EOF in block 43 of relation base/16386/2673 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 50 of relation base/16386/2674 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 80 of relation base/16386/1255 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] CREATE EXTENSION
    - execute /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS btree_gist"

Then I remounted "/mnt/g/" to kNFS and tried again. It works, not very fast but works.

The exact command to reproduce the issue:

The full output of the command that failed:

postgresql_extension[btree_gist] action enable
- postgresql_query[enable btree_gist extension] action run
  - execute[enable btree_gist extension (postgresql)] action run [execute] ERROR: unexpected data beyond EOF in block 43 of relation base/16386/2673 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 50 of relation base/16386/2674 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 80 of relation base/16386/1255 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] CREATE EXTENSION
    - execute /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS btree_gist"

Expected results: running container with postgres inside

Mandatory info: - The output of the gluster volume info command: root@u18-1:~# gluster volume info

Volume Name: vol01 Type: Distribute Volume ID: 46f50403-d590-46ed-a95d-70cce3377344 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gluster01:/glusterfs/distributed Brick2: gluster02:/glusterfs/distributed Options Reconfigured: storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on

Volume Name: vol02 Type: Replicate Volume ID: f1b50483-6964-4be6-9c8d-9beb86aa3da5 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster01:/glusterfs/replicated Brick2: gluster02:/glusterfs/replicated Options Reconfigured: features.cache-invalidation: on auth.allow: 1.2.3.4,192.168.122.1 features.quota-deem-statfs: on features.inode-quota: on features.quota: on performance.client-io-threads: off nfs.disable: on transport.address-family: inet storage.fips-mode-rchecksum: on

- The output of the `gluster volume status` command: root@u18-1:~# gluster volume status Status of volume: vol01 Gluster process TCP Port RDMA Port Online Pid

Brick gluster01:/glusterfs/distributed 49152 0 Y 577
Brick gluster02:/glusterfs/distributed 49152 0 Y 665

Task Status of Volume vol01

There are no active volume tasks

Status of volume: vol02 Gluster process TCP Port RDMA Port Online Pid

Brick gluster01:/glusterfs/replicated 49153 0 Y 588
Brick gluster02:/glusterfs/replicated 49153 0 Y 707
Self-heal Daemon on localhost N/A N/A Y 610
Quota Daemon on localhost N/A N/A Y 599
Self-heal Daemon on gluster02 N/A N/A Y 931
Quota Daemon on gluster02 N/A N/A Y 854

Task Status of Volume vol02

There are no active volume tasks

- The output of the gluster volume heal command:

root@u18-1:~# gluster volume heal vol02 statistics (?) Gathering crawl statistics on volume vol02 has been successful

Crawl statistics for brick no 0 Hostname of brick gluster01

Crawl statistics for brick no 1 Hostname of brick gluster02

**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/ No errors

**- Is there any crash ? Provide the backtrace and coredump No

Additional info: I know about https://docs.gluster.org/en/latest/Install-Guide/Overview/ "Gluster does not support so called “structured data”, meaning live, SQL databases. Of course, using Gluster to backup and restore the database would be fine"

Does this mean that postgres will not work on glusterfs and there is no any workaround ?

Also I used only two nodes for replication but should use 3. I tried also with distributed volume and got the same result. I found an old issue (2018) but have no link to this issue.

- The operating system / glusterfs version: 8.3

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration

amarts commented 3 years ago

there are options for databases on glusterfs. Please turn off performance translators (gluster volume set vol01 group db-workload) and try.

warmanton commented 3 years ago

there are options for databases on glusterfs. Please turn off performance translators (gluster volume set vol01 group db-workload) and try.

Just tried the same installation with Your recommendation. Does not work. Now I see no that errors but postgres cannot start or restart. After that grafana failed to restart

B.t.w. where can I find a documentation about "group db-workload" and other groups ? Please give me some links.

Notes about gitlab in docker: this is a bundle of services i.e. about 10 services inside one gitlab container. This is a not docker way but distributes their product in a single container. I use gitlab as a marker -- if gitlab will work on gluster then almost all other service should work on this kind of permanent storage (mounted gluster volume).

warmanton commented 3 years ago

B.t.w. steps to reproduce the issue:

docker run  \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 9022:22 \
  --name gitlab \
  --restart always \
  --volume /mnt/n/gitlab/config:/etc/gitlab \
  --volume /mnt/n/gitlab/logs:/var/log/gitlab \
  --volume /mnt/n/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:13.5.7-ce.0

where /mnt/n/ is a mount point for GlusterFS. gitlab version may differ. It failed on postgresql which is inside of gitlab.

jicki commented 3 years ago

B.t.w. steps to reproduce the issue:
docker run  \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 9022:22 \
  --name gitlab \
  --restart always \
  --volume /mnt/n/gitlab/config:/etc/gitlab \
  --volume /mnt/n/gitlab/logs:/var/log/gitlab \
  --volume /mnt/n/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:13.5.7-ce.0
where /mnt/n/ is a mount point for GlusterFS. gitlab version may differ. It failed on postgresql which is inside of gitlab.

I also encountered this problem

Is there a solution, thank~

thewhitewizard commented 3 years ago

Hi guys,

I also encoutered this problem with Swarm+Postgres+gluster. I fix it just now :

gluster volume set gvol0 performance.stat-prefetch off

https://access.redhat.com/solutions/3673761

danielzhanghl commented 2 years ago

found the required parameters when using glusterfs to support database application:

https://docs.openshift.com/container-platform/3.10/scaling_performance/optimizing_on_glusterfs_storage.html

The performance translators for GlusterFS, as mentioned above, are already part of the database profile delivered with the latest converged mode images.

Test Results

For Postgresql SQL databases, GlusterFS and gluster-block showed approximately the same performance results. For MongoDB noSQL databases, gluster-block performed better. Therefore, use gluster-block based storage for MongoDB noSQL databases.

[](https://docs.openshift.com/container-platform/3.10/scaling_performance/optimizing_on_glusterfs_storage.html#Test-Results)Database

stale[bot] commented 1 year ago

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

stale[bot] commented 1 year ago

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.

MPParsley commented 1 year ago

Ran into this issue when running a simple sql import on a default Openshift postgress database.

aguinaldoabbj commented 1 year ago

Ran into this issue when running a simple sql import on a default Openshift postgress database.

And have you managed to solve your problem? I'm struggling with the very same problem of the issuer, but the pointed solution doesn't work for me.

MPParsley commented 1 year ago

Alas, our solution was to move to Trident StorageClass.

aguinaldoabbj commented 1 year ago

Alas, our solution was to move to Trident StorageClass.

Could you please link me up ?

aguinaldoabbj commented 1 year ago

Description of problem: Trying to get persistent storage for docker swam. Testing on single node swarm + two nodes with Gluster server. Trying to deploy gitlab instance with thee bind mounts targeted on GlusterFS mount point. Volumes look like thes:
volumes:
  - /mnt/g/gitlab/data:/var/opt/gitlab
  - /mnt/g/gitlab/logs:/var/log/gitlab
  - /mnt/g/gitlab/config:/etc/gitlab
Got failed docker container with messages like these:

postgresql_extension[btree_gist] action enable

postgresql_query[enable btree_gist extension] action run

execute[enable btree_gist extension (postgresql)] action run [execute] ERROR: unexpected data beyond EOF in block 43 of relation base/16386/2673 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 50 of relation base/16386/2674 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 80 of relation base/16386/1255 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] CREATE EXTENSION

execute /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS btree_gist"

Then I remounted "/mnt/g/" to kNFS and tried again. It works, not very fast but works.

The exact command to reproduce the issue:

The full output of the command that failed:

postgresql_extension[btree_gist] action enable

postgresql_query[enable btree_gist extension] action run

execute[enable btree_gist extension (postgresql)] action run [execute] ERROR: unexpected data beyond EOF in block 43 of relation base/16386/2673 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 50 of relation base/16386/2674 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] ERROR: unexpected data beyond EOF in block 80 of relation base/16386/1255 HINT: This has been seen to occur with buggy kernels; consider updating your system. [execute] CREATE EXTENSION

execute /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS btree_gist"

Expected results: running container with postgres inside

Mandatory info: - The output of the gluster volume info command: root@u18-1:~# gluster volume info

Volume Name: vol01 Type: Distribute Volume ID: 46f50403-d590-46ed-a95d-70cce3377344 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gluster01:/glusterfs/distributed Brick2: gluster02:/glusterfs/distributed Options Reconfigured: storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on

Volume Name: vol02 Type: Replicate Volume ID: f1b50483-6964-4be6-9c8d-9beb86aa3da5 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster01:/glusterfs/replicated Brick2: gluster02:/glusterfs/replicated Options Reconfigured: features.cache-invalidation: on auth.allow: 1.2.3.4,192.168.122.1 features.quota-deem-statfs: on features.inode-quota: on features.quota: on performance.client-io-threads: off nfs.disable: on transport.address-family: inet storage.fips-mode-rchecksum: on

- The output of the gluster volume status command:

root@u18-1:~# gluster volume status Status of volume: vol01 Gluster process TCP Port RDMA Port Online Pid Brick gluster01:/glusterfs/distributed 49152 0 Y 577 Brick gluster02:/glusterfs/distributed 49152 0 Y 665

Task Status of Volume vol01

There are no active volume tasks

Status of volume: vol02

Gluster process TCP Port RDMA Port Online Pid Brick gluster01:/glusterfs/replicated 49153 0 Y 588 Brick gluster02:/glusterfs/replicated 49153 0 Y 707 Self-heal Daemon on localhost N/A N/A Y 610 Quota Daemon on localhost N/A N/A Y 599 Self-heal Daemon on gluster02 N/A N/A Y 931 Quota Daemon on gluster02 N/A N/A Y 854

Task Status of Volume vol02

There are no active volume tasks

- The output of the gluster volume heal command:

root@u18-1:~# gluster volume heal vol02 statistics (?)

Gathering crawl statistics on volume vol02 has been successful

Crawl statistics for brick no 0

Hostname of brick gluster01 Crawl statistics for brick no 1 Hostname of brick gluster02

**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/ No errors

**- Is there any crash ? Provide the backtrace and coredump No

Additional info: I know about https://docs.gluster.org/en/latest/Install-Guide/Overview/ "Gluster does not support so called “structured data”, meaning live, SQL databases. Of course, using Gluster to backup and restore the database would be fine"

Does this mean that postgres will not work on glusterfs and there is no any workaround ?

Also I used only two nodes for replication but should use 3. I tried also with distributed volume and got the same result. I found an old issue (2018) but have no link to this issue.@

- The operating system / glusterfs version: 8.3

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration

@warmanton Sorry to necrobumping. Have you managed to use gitlab on glusterfs ?

MPParsley commented 1 year ago

Could you please link me up ?

https://netapp-trident.readthedocs.io/en/stable-v19.01/kubernetes/index.html

gluster / glusterfs