Affect the decompression speed

[lateautumn233]

Use pv

Do not use

I don't know why it has such a big impact

[a-j-wood]

Hello What kind of system is this running on? (architecture and distribution)

Is splice() available, and if so, does the "-C" option make any difference?

Does passing different values to "-B" make any difference?

[lateautumn233]

Hello What kind of system is this running on? (architecture and distribution)

Is splice() available, and if so, does the "-C" option make any difference?

Does passing different values to "-B" make any difference?

I will conduct more tests

[lateautumn233]

splice() available

pv test.tar.zst -c | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:00:55 [37.3MiB/s] [=======================================>] 100%

tar "-I zstd -T0" -axf test.tar.zst --checkpoint-action="ttyout=Hit %T\r"
Hit R: 2088028160 (2.0GiB, 232MiB/s))

pv -B How should I test it?

[a-j-wood]

It's "pv -C" / "pv --no-splice" to turn off splice, not "pv -c" (which is "pv --cursor").

With "pv -B", try different values to see how they affect the transfer rate. For example "pv -B 1024", "pv -B 1048576", etc.

The manual says that using "-B" implies "-C" automatically, so you don't need to try both, assuming you're using PV 1.6.20.

[lateautumn233]

It's "pv -C" / "pv --no-splice" to turn off splice, not "pv -c" (which is "pv --cursor").

With "pv -B", try different values to see how they affect the transfer rate. For example "pv -B 1024", "pv -B 1048576", etc.

The manual says that using "-B" implies "-C" automatically, so you don't need to try both, assuming you're using PV 1.6.20.

uname -a

Linux localhost 5.10.136-android12-9-00021-g821df8f5bd36-ab9585204 #1 SMP PREEMPT Thu Feb 9 13:08:46 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

pv --no-splice

time pv test.tar.zst --no-splice | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.00GiB 0:01:01 [19.9MiB/s] [=======================================> ] 99% ETA 0:00:00

2.01GiB 0:01:01 [33.5MiB/s] [=======================================>] 100%

________________________________________________________
Executed in   61.46 secs    fish           external
   usr time   11.07 secs    1.25 millis   11.07 secs
   sys time   77.15 secs    0.96 millis   77.15 secs

pv -C

time pv test.tar.zst -C | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.30GiB 0:00:41 [39.6MiB/s] 1.87GiB 0:00:58 [5.92MiB/s] [=====================================>   ]
93% ETA 0:00:04

2.01GiB 0:01:12 [28.2MiB/s] [=======================================>] 100%

________________________________________________________
Executed in   73.16 secs    fish           external
   usr time   12.87 secs    0.00 micros   12.87 secs
   sys time   76.53 secs  942.00 micros   76.53 secs

[a-j-wood]

Sorry I wasn't clear about -B implying -C, please try with -B for various values such as "-B 1024" or "-B 1048576". That may improve performance.

Can you recommend a way I can build or access a similar system to yours, to reproduce the behaviour? Does it do the same on other aarch64 systems such as a Pi 4 for example?

[lateautumn233]

Sorry I wasn't clear about -B implying -C, please try with -B for various values such as "-B 1024" or "-B 1048576". That may improve performance.

Can you recommend a way I can build or access a similar system to yours, to reproduce the behaviour? Does it do the same on other aarch64 systems such as a Pi 4 for example?

I'm sorry I forgot to tell you This is an Android device You can test on termux

[a-j-wood]

Can you try with "-B 1024", "-B 4096", "-B 65536", "-B 1048576", see what difference each of those make?

[lateautumn233]

root@localhost ~# ./pv -B 1024 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.26GiB 0:00:40 [29.5MiB/s] [========================>                ] 62% ETA 0:00:23
2.01GiB 0:01:07 [-6.66e+24yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 4096 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:14 [-1.30e+24yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 65536 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:18 [-5.07e+25yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 1048576 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:15 [5.59MiB/s] [=======================================>] 100%            

root@localhost ~# strace -o pv.log ./pv test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.89GiB 0:00:24 [9.76MiB/s] [=====================================>   ] 94% ETA 0:00:01
1.90GiB 0:00:26 [8.31MiB/s] [=====================================>   ] 94% ETA 0:00:01
1.92GiB 0:00:28 [10.7MiB/s] [=====================================>   ] 95% ETA 0:00:01
2.01GiB 0:00:36 [-2.95e+24yiB/s] [==================================>] 100%

I don't understand

[a-j-wood]

Which version of pv was that? ("./pv -V") The nonsensical negative final value should have gone away in 1.7.0.

On your last example you did "strace -o pv.log ./pv test.tar.zst | ..." and it was much faster, is it still much faster if you remove the strace so it's just "./pv test.tar.zst | ..."?

[lateautumn233]

Yes, it's much faster after I added stace

root@localhost ~ [128]# ./pv -V                                                        
pv 1.6.20
- Copyright 2015 Andrew Wood <andrew.wood@ivarch.com>

Web site: http://www.ivarch.com/programs/pv.shtml

This program is free software, and is being distributed under the
terms of the Artistic License 2.0.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@localhost ~# ./pv test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:17 [5.53MiB/s] [=======================================>] 100%

[a-j-wood]

So it is faster with strace. That is very strange.

Can you try with pv 1.7.0 please?

If you repeat your tests with "-T1" passed to zstd instead of "-T0", what difference does that make?

[lateautumn233]

root@localhost ~# ./pvnew -V
pv 1.7.0 
- Copyright 2015 Andrew Wood <andrew.wood@ivarch.com>

Web site: http://www.ivarch.com/programs/pv.shtml

This program is free software, and is being distributed under the
terms of the Artistic License 2.0.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@localhost ~# ./pvnew test.tar.zst  | tar "-I zstd -T1" -x                         
2.01GiB 0:00:48 [42.2MiB/s] [=======================================>] 100%

[a-j-wood]

This project has moved to Codeberg so the issue is open there instead: https://codeberg.org/a-j-wood/pv/issues/59