search

stratis-storage / project

A holding place for issues that affect more than one repository in the project
4 stars 0 forks source link

[BUG?] the requested filesystems already exist; no action taken #686

Closed bsdooby closed 4 months ago

bsdooby commented 4 months ago

Context: on Slackware (-current), after successful installation of stratis (latest) from source (adding all required dependencies), I want to create a filesystem on my test pool (2 x 7 GB USB keys)

Want: create new filesystem

Get: from stratisd: the requested filesystems already exist; no action taken

Observation: no fs is created on the pool, as can be investigated by stratis fs list mypool

What could cause this?

mulkieran commented 4 months ago

Please post the following if possible:

  1. Command you invoked to create the pool.
  2. Output of stratis report.
  3. Output of stratis pool list --stopped.
  4. Output of stratis report managed_objects_report (this will be D-Bus info, you'll probably have to upload it).
  5. Output of lsblk.

Thanks!

mulkieran commented 4 months ago

Since the error is generated by stratis not stratisd we know that stratis is inferring from the information on the D-Bus that a filesystem with the name you specified already exists. stratis report will allow us to get at the ground truth of stratisd's picture of the world, stratis report managed_objects_report will allow us to get an understanding of what is on the D-Bus. Also, stratis is latest from source, if that is from git, can you give us a commit hash? If not, can you be as specific as possible about the source? What version of stratisd are you using?

bsdooby commented 4 months ago

I retested with only one physical disk (USB key).

$ stratis daemon version
3.7.0

Output of stratis report.:

{
    "name_to_pool_uuid_map": {},
    "partially_constructed_pools": [],
    "path_to_ids_map": {},
    "pools": [
        {
            "available_actions": "fully_operational",
            "blockdevs": {
                "cachedevs": [],
                "datadevs": [
                    {
                        "blksizes": "base: BLKSSSZGET: 512 bytes, BLKPBSZGET: 512 bytes, crypt: None",
                        "in_use": true,
                        "path": "/dev/sdb",
                        "size": "15728640 sectors",
                        "uuid": "7a53520f-d0b8-47a6-a92d-e35414905c58"
                    }
                ]
            },
            "filesystems": [],
            "fs_limit": 100,
            "name": "mypool",
            "uuid": "105c1762-a5a1-478d-8115-a9f620905cbb"
        }
    ],
    "stopped_pools": []
}

Output of stratis pool list --stopped:

$ stratis pool list --stopped
Name   UUID   # Devices   Key Description   Clevis

Output of stratis report managed_objects_report (this will be D-Bus info, you'll probably have to upload it). (see attachment): managed_objects_report.txt

Output of lsblk (abbridged):

...
sdb                                                                       8:16   1   7.5G  0 disk    
└─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-physical-originsub 252:0    0   7.5G  0 stratis 
  ├─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-flex-thinmeta    252:1    0     6M  0 stratis 
  │ └─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-thinpool-pool  252:3    0     7G  0 stratis 
  ├─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-flex-thindata    252:2    0     7G  0 stratis 
  │ └─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-thinpool-pool  252:3    0     7G  0 stratis 
  └─stratis-1-private-105c1762a5a1478d8115a9f620905cbb-flex-mdv         252:4    0   512M  0 stratis
...
mulkieran commented 4 months ago

Ok! What was the name of the filesystem you were expecting to create?

bsdooby commented 4 months ago

client version: 

$ stratis --version
3.6.0

source versions (from git): stratis-cli: c126fc24f9d707f360b... stratisd: eca8bec033ec94d92b0...

bsdooby commented 4 months ago

Ok! What was the name of the filesystem you were expecting to create?

(?) The standard fs that is used for Stratis (XFS); I do not need to indicate this, right?

mulkieran commented 4 months ago

Ok! What was the name of the filesystem you were expecting to create?

(?) The standard fs that is used for Stratis (XFS); I do not need to indicate this, right?

When you create a new filesystem, you have to give it a name. The command is: stratis filesystem create <poolname> <filesystem name> [options]. So, can you tell me the exact command that you were using to create the filesystem?

mulkieran commented 4 months ago

By the way, I was mistaken formerly. The error is generated by stratisd, not stratis-cli. So the whole thing, the precise command you issued and the precise error reported would be a help.

bsdooby commented 4 months ago

I issued: stratis fs create mypool myfs and got:

Execution failed: stratisd reported that it did not execute every action that it would have been expected to execute as a result of the command that you requested: Expected to create the specified filesystems in pool mypool but stratisd reports that it did not actually create some or all of the filesystems requested

report from stratisd --with-log=info:

...stratisd::dbus_api::pool::pool_3_6::methods] The requested filesystems already exist; no action taken)

bsdooby commented 4 months ago

My platform is a non-systemd one; Slackware64 -current w/ thin-provisioning-tools 0.9

mulkieran commented 4 months ago

I issued: stratis fs create mypool myfs and got:

Execution failed: stratisd reported that it did not execute every action that it would have been expected to execute as a result of the command that you requested: Expected to create the specified filesystems in pool mypool but stratisd reports that it did not actually create some or all of the filesystems requested

report from stratisd --with-log=info:

...stratisd::dbus_api::pool::pool_3_6::methods] The requested filesystems already exist; no action taken)

Ok!

The error message on the command-line is from stratis, stratisd reported to it that it had not created any filesystems as part of the return value of the D-Bus method, and stratis reported that it found that very surprising.

Can you give more output from the stratisd logs up to the one log entry you quoted? That entry is emitted exactly when the D-Bus method returns, so log entries that precede that one should be valuable. Also, please set the log level to debug instead of info.

jbaublitz commented 4 months ago

@mulkieran I actually saw this error in my metadata rework PR but need to look back to see exactly what was happening. Either I had a mismatched version of stratis-cli and stratisd or there was a bug in the engine. I can't remember which it was, and I can't reproduce it on the master branch, but I'll take a look again.

jbaublitz commented 4 months ago

Aha! I think I tracked it down. What I think is happening is that stratis-cli is not updated. Because the pre-3.6 version of stratis-cli and post-3.6 version of stratis-cli both send an array as the only parameter, our D-Bus layer allows us to send an incorrectly formatted array from the pre-3.6 version of stratis-cli to the 3.6 version of stratisd. This results in an empty set of filesystem specs being passed to the engine. This trips of stratis-cli because due to being idempotent, the engine reports that creating no filesystems was successful.

mulkieran commented 4 months ago

Bingo! This is a known D-Bus bug: https://github.com/stratis-storage/project/issues/11 . @bsdooby The best thing for you to do is to use a newer version of stratis-cli and that will resolve your immediate problem. You will generally be better off if you check out the source from a specific tag.

bsdooby commented 4 months ago 
# ./stratisd --log-level=debug
[2024-02-06T20:12:20Z INFO  stratisd::stratis::run] stratis daemon version 3.7.0 started
[2024-02-06T20:12:20Z INFO  stratisd::stratis::run] Using StratEngine
[2024-02-06T20:12:20Z INFO  stratisd::engine::strat_engine::liminal::identify] Beginning initial search for Stratis block devices
[2024-02-06T20:12:20Z INFO  stratisd::engine::strat_engine::liminal::identify] Stratis block device with Stratis pool UUID: "105c1762-a5a1-478d-8115-a9f620905cbb", Stratis device UUID: "7a53520f-d0b8-47a6-a92d-e35414905c58", device number: "8:16", devnode: "/dev/sdb" discovered during initial search
[2024-02-06T20:12:20Z INFO  stratisd::engine::strat_engine::liminal::device_info] Device information Stratis device description: Stratis pool UUID: "105c1762-a5a1-478d-8115-a9f620905cbb", Stratis device UUID: "7a53520f-d0b8-47a6-a92d-e35414905c58", device number: "8:16", devnode: "/dev/sdb" discovered and inserted into the set for its pool UUID
[2024-02-06T20:12:20Z INFO  stratisd::engine::strat_engine::liminal::liminal] Pool with name "mypool" and UUID "105c1762-a5a1-478d-8115-a9f620905cbb" set up
[2024-02-06T20:12:20Z INFO  stratisd::stratis::ipc_support::dbus_support] D-Bus API is available
[2024-02-06T20:12:20Z WARN  stratisd::dbus_api::tree] A pool property was changed in the engine but no pool with the corresponding UUID, 105c1762-a5a1-478d-8115-a9f620905cbb, could be found in the D-Bus layer
[2024-02-06T20:12:28Z INFO  stratisd::dbus_api::pool::pool_3_6::methods] The requested filesystems already exist; no action taken
bsdooby commented 4 months ago

@mulkieran Which tag of the stratis-cli would be the best (current) working version?

mulkieran commented 4 months ago

@bsdooby I would recommend v3.6.0.

bsdooby commented 4 months ago

@mulkieran Seems to work. How cool is this :) I will test some more and report back, OK? Please keep this issue open.

bsdooby commented 4 months ago

from dmesg I see the following (but mounting the newly created fs seems to work):

[Tue Feb  6 21:19:46 2024] device-mapper: thin: 252:3: reached low water mark for data device: sending event.
[Tue Feb  6 21:21:01 2024] udevd[6854]: failed to execute '/lib/udev/stratis-base32-decode' 'stratis-base32-decode STRATIS_POOL_NAME NV4XA33PNQ': No such file or directory
[Tue Feb  6 21:21:01 2024] udevd[6855]: failed to execute '/lib/udev/stratis-base32-decode' 'stratis-base32-decode STRATIS_FS_NAME NV4XGZQ': No such file or directory
[Tue Feb  6 21:23:09 2024] udevd[6935]: failed to execute '/lib/udev/stratis-base32-decode' 'stratis-base32-decode STRATIS_POOL_NAME NV4XA33PNQ': No such file or directory
[Tue Feb  6 21:23:09 2024] udevd[6936]: failed to execute '/lib/udev/stratis-base32-decode' 'stratis-base32-decode STRATIS_FS_NAME NV4WM4Y': No such file or directory
[Tue Feb  6 21:24:26 2024] XFS (dm-5): Mounting V5 Filesystem dc21fea4-c822-46f0-b08b-05f71147d97a
[Tue Feb  6 21:24:27 2024] XFS (dm-5): Ending clean mount
jbaublitz commented 4 months ago

My guess is that you need to set an environment variable during the build/install phase to point to /lib/udev instead of the standard directory for Fedora which is /usr/lib/udev.

bsdooby commented 4 months ago

@jbaublitz what would that flag be? It seems to work, ignoring that "errror"

jbaublitz commented 4 months ago

If you override either $PREFIX or $UDEVDIR, that should do it, but take a look at the environment variables at the beginning of our Makefile and how they're used in various targets. I'm not sure which you'd rather use because I'm unfamiliar with Slackware's directory hierarchy. $PREFIX applies to all ~/usr/lib targets~ correction: /usr targets while $UDEVDIR would just apply to the udev parts that are installed.

bsdooby commented 4 months ago

@jbaublitz I will have a look; thank you en tout can...

bsdooby commented 4 months ago

UDEVDIR does not seem to exist and setting it has no effect (the error message still is the same): udevd[9477]: failed to execute '/lib/udev/stratis-base32-decode' 'stratis-base32-decode STRATIS_POOL_NAME NV4XA33PNQ': No such file or directory

jbaublitz commented 4 months ago

Can you please provide the command you used to install?

bsdooby commented 4 months ago

export UDEVDIR=/lib/udev followed by make build followed by make install which gives mkdir -p /lib/udev/rules.d /usr/bin/install -Dpm0644 -t /lib/udev/rules.d udev/61-stratisd.rules mkdir -p /usr/share/man/man8 /usr/bin/install -Dpm0644 -t /usr/share/man/man8 docs/stratisd.8 /usr/bin/install: cannot stat 'docs/stratisd.8': No such file or directory make: *** [Makefile:285: install-man-cfg] Error 1

bsdooby commented 4 months ago

I was under the impression that make build builds all necessary tools, but there are other make targets for the udev tools (like stratis-base32-decode, stratis-str-cmp); why are they needed, and why a special build target for these tools?

mulkieran commented 4 months ago

@bsdooby Try make build-all to setup for make install. Those small executables that you mention are there to support some udev rules which are also installed via make install.

bsdooby commented 4 months ago

Yes, I already did that (see my latest post). I will observe dmesg -w ... stay tuned.

[off topic] How would you define the USP of stratis (compared to ZFS and/or Btrfs)?

On Wed, Feb 7, 2024 at 6:26 PM mulkieran @.***> wrote:

@bsdooby https://github.com/bsdooby Try make build-all to setup for make install. Those small executables that you mention are there to support some udev rules which are also installed via make install.

— Reply to this email directly, view it on GitHub https://github.com/stratis-storage/project/issues/686#issuecomment-1932533475, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACIWLMNQ2SSOH33PVK2Z5H3YSO2KTAVCNFSM6AAAAABC36IJCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGUZTGNBXGU . You are receiving this because you were mentioned.Message ID: @.***>

bsdooby commented 4 months ago

Another issue that I have: df -h shows wrong storage details on a pool/filesystem of (only) 7GB: Size Used Avail Use% Mounted on ... dev/mapper/stratis-1-57d589540cbb4529b19d3dfcd34432f4-thin-fs-3dbf4970c6234b24aadfcfbca55cdad8 1.0T 20G 1004G 2% /mnt/stratis and stratis filesystem list

Pool Filesystem Total / Used / Free / Limit Created Device UUID pool stratis 1 TiB / 545 MiB / 1023.47 GiB / None Feb 07 2024 19:49 /dev/stratis/pool/stratis 3dbf4970-c623-4b24-aadf-cfbca55cdad8

jbaublitz commented 4 months ago

That is actually correct information. Thin provisioning allows filesystems to be larger than the total physical size of the storage available and filesystems are set to 1 TiB by default. If you're concerned about exhausting the physical space on your pool, I'd recommend either setting a size and size limit lower than the physical space on your pool (stratis fs create --size --size-limit) or setting the no overprovisioning mode on the pool to avoid the total size of filesystems from exceeding the available physical storage (stratis pool overprovision).

bsdooby commented 4 months ago

Ah OK, makes sense. I will play around with --size and --size-limits (and to not let the pool overprovision).

On Wed, Feb 7, 2024 at 8:10 PM John Baublitz @.***> wrote:

That is actually correct information. Thin provisioning allows filesystems to be larger than the total physical size of the storage available and filesystems are set to 1 TiB by default. If you're concerned about exhausting the physical space on your pool, I'd recommend either setting a size and size limit lower than the physical space on your pool (stratis fs create --size --size-limit) or setting the no overprovisioning mode on the pool to avoid the total size of filesystems from exceeding the available physical storage (stratis pool overprovision).

— Reply to this email directly, view it on GitHub https://github.com/stratis-storage/project/issues/686#issuecomment-1932698368, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACIWLMNMBEWL65SJD3BAWADYSPGTBAVCNFSM6AAAAABC36IJCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGY4TQMZWHA . You are receiving this because you were mentioned.Message ID: @.***>

bsdooby commented 4 months ago

I think all is good now w/ the help and information you provided.