Unable to start copying

zollty commented 1 year ago

Expected behavior

mc cp --recursive local/tianditu .

Actual behavior

mc: Unable to start copying. We encountered an internal error, please try again.

[root@minio-new personal]# mc cp --recursive local/tianditu .
mc: <ERROR> Unable to start copying. We encountered an internal error, please try again.
...00/tianditu/2021/17/43982_208745:  672.59 MiB / 672.59 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 3.26 MiB/s 3m26s [root@minio-new personal]# 
[root@minio-new personal]# 
[root@minio-new personal]# 
[root@minio-new personal]# rm -rf tianditu/
[root@minio-new personal]# mc cp --recursive local/tianditu .
mc: <ERROR> Unable to start copying. We encountered an internal error, please try again.
...00/tianditu/2021/17/43813_208636:  50.80 MiB / 50.80 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 1.02 MiB/s 49s [root@minio-new personal]# 
[root@minio-new personal]# 
[root@minio-new personal]# 
[root@minio-new personal]# rm -rf tianditu/
[root@minio-new personal]# mc cp --recursive local/tianditu .
mc: <ERROR> Unable to start copying. We encountered an internal error, please try again.
...00/tianditu/2021/17/43852_208740:  128.29 MiB / 128.29 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 2.67 MiB/s 47s [root@minio-new personal]# 
[root@minio-new personal]#

Steps to reproduce the behavior

mc --version

mc version RELEASE.2021-10-07T04-19-58Z

System information

CentOS Linux release 7.6.1810 (Core) Linux 3.10.0-957.el7.x86_64

harshavardhana commented 1 year ago

Please provide --debug output @zollty

zollty commented 1 year ago

mc --debug：

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message><Resource>/tianditu/</Resource><Region>cn-south-cq-1</Region><RequestId>172185218FBC660F</RequestId><HostId>9ae4b8e1-1d3f-4123-ad16-dba2f606b982</HostId></Error>mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: DigiCert Inc
mc: <DEBUG>  >> Expires: 2022-11-26 23:59:59 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: DigiCert Inc
mc: <DEBUG>  >> Expires: 2027-11-27 12:46:10 +0000 UTC
mc: <DEBUG> Response Time:  2.766146745s

...00/tianditu/2021/17/43813_208636:  50.80 MiB / 50.80 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 1.38 MiB/s mc: <DEBUG> GET /tianditu/?continuation-token=MjAyMS8xNy80MzgxM18yMDg2MzZbbWluaW9fY2FjaGU6djIsaWQ6YjVmNTQ2YjMtOWMxZi00Zjc2LWI1ODgtZGI2YzVhZDIxZjIxLHA6MCxzOjBd&delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1
Host: pos.paramland.cn:9000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z
Authorization: AWS4-HMAC-SHA256 Credential=posadmin/20221026/cn-south-cq-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221026T044839Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 500 Internal Server Error
Content-Length: 303
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Wed, 26 Oct 2022 04:48:41 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: cn-south-cq-1
X-Amz-Request-Id: 172185223496D08F
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

minio server log

API: ListObjectsV2
Time: 04:33:53:0
DeploymentID: 9ae4b8e1-1d3f-4123-ad16-dba2f606b982
RequestID: 17218453EC47F36F
RemoteHost: 127.0.0.1
Host: pos.paramland.cn:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z
Error: Drive /mnt/data4, path (.minio.sys/buckets/tianditu/.metacache/8fa37872-c939-4b66-a9a6-5360db4d1a4f/block-0.s2) returned an error (unexpected EOF) (*fmt.wrapError)
internal/logger/logonce.go:54:logger.(*logOnceType).logOnceIf()
internal/logger/logonce.go:94:logger.LogOnceIf()
cmd/erasure-metadata-utils.go:139:cmd.readAllFileInfo.func1()
internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

API: SYSTEM
Time: 04:33:53:0
DeploymentID: 9ae4b8e1-1d3f-4123-ad16-dba2f606b982
Error: Disk: /mnt/data1 -> .minio.sys/buckets/tianditu/.metacache/8fa37872-c939-4b66-a9a6-5360db4d1a4f/block-0.s2/9ff599aa-bc36-4824-9a82-43fe4a54f4cd/part.1 - content hash does not match - expected 49035da364c15dc6c68e9fdb9257875cde5ccce77b3292770349ca58ceb6eadd, got 0b6823c6817c16e4e78a313eba8eacd929f2c71e159499886d66b7edf2a5f249 (*errors.errorString)
cmd/bitrot-streaming.go:183:cmd.(*streamingBitrotReader).ReadAt()
cmd/erasure-decode.go:165:cmd.(*parallelReader).Read.func1()

harshavardhana commented 1 year ago

There is some issue on the server side here @zollty which version of MinIO are you running?

zollty commented 1 year ago

Just now, I tried again and executed the command successfully：（I didn't make any changes, just tried several times）

[root@minio-new personal]# mc cp --recursive local/tianditu .
(....omit)
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 53485
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: image/jpeg
Date: Wed, 26 Oct 2022 05:03:41 GMT
Etag: "f5d88422f17143c1b2b5020081e96129"
Last-Modified: Wed, 28 Sep 2022 18:20:02 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Bucket-Region: cn-south-cq-1
X-Amz-Request-Id: 172185F469E61D8A
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: DigiCert Inc
mc: <DEBUG>  >> Expires: 2022-11-26 23:59:59 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: DigiCert Inc
mc: <DEBUG>  >> Expires: 2027-11-27 12:46:10 +0000 UTC
mc: <DEBUG> Response Time:  71.086395ms

...00/tianditu/2021/17/44286_208898:  1.35 GiB / 1.35 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 3.06 MiB/s 7m30s [root@minio-new personal]# 
[root@minio-new personal]#

zollty commented 1 year ago

minio version RELEASE.2021-11-09T03-21-45Z

The minio and mc-client are on the same server.

harshavardhana commented 1 year ago

Yeah this was a bug in Listing code in that version of server, you must upgrade to latest releases.

minio / mc