filecoin-project / lily

capturing on-chain state for the filecoin network
Other
49 stars 45 forks source link

Data dump: invalid byte sequence for encoding "UTF8": 0xa1 for market_deal_proposals #843

Closed iand closed 2 years ago

iand commented 2 years ago

Describe the bug:

Originally reported in #824 for dates 2020-09-08, 2020-09-27

ERROR: invalid byte sequence for encoding "UTF8": 0xa1

Steps to Reproduce:

Lily Version: vx.x.x

iand commented 2 years ago

2020-09-08 is height range 40560-43439 2020-09-27 is 95280-98159

iand commented 2 years ago

There are some unusual encoded strings in the label field for s3://lily-data/data/95280__98159/market_deal_proposals.csv

95385,173870,bafy2bzacecw7oxgjwanokadrpz2qkvufkerv7gehm2jb7mppg5pmuau3qezcu,262144,260096,98260,1139144,f022013,f09833,0,0,122070,baga6ea4seaqnxgztgkfpklt4a2m7jlzdvhrfrc2nexwptqyzcwufs4pqy2qtegy,false,"<A1>epcids<81><D8>*X#^@^R z
<95><A5>Y*<AC>G5^N<80>[    亽^V<F0><FF><EE>$<C7>^W^@zX<8C>""|}<E5>֝"
95385,173871,bafy2bzacecw7oxgjwanokadrpz2qkvufkerv7gehm2jb7mppg5pmuau3qezcu,262144,260096,98260,1138855,f022013,f02613,0,0,122070,baga6ea4seaqnxgztgkfpklt4a2m7jlzdvhrfrc2nexwptqyzcwufs4pqy2qtegy,false,"<A1>epcids<81><D8>*X#^@^R z
<95><A5>Y*<AC>G5^N<80>[    亽^V<F0><FF><EE>$<C7>^W^@zX<8C>""|}<E5>֝"
95385,173872,bafy2bzacecw7oxgjwanokadrpz2qkvufkerv7gehm2jb7mppg5pmuau3qezcu,262144,260096,98260,1140153,f022013,f02490,0,0,122070,baga6ea4seaqnxgztgkfpklt4a2m7jlzdvhrfrc2nexwptqyzcwufs4pqy2qtegy,false,"<A1>epcids<81><D8>*X#^@^R z
<95><A5>Y*<AC>G5^N<80>[    亽^V<F0><FF><EE>$<C7>^W^@zX<8C>""|}<E5>֝"
95385,173873,bafy2bzacecw7oxgjwanokadrpz2qkvufkerv7gehm2jb7mppg5pmuau3qezcu,262144,260096,98260,1139947,f022013,f02301,0,0,122070,baga6ea4seaqnxgztgkfpklt4a2m7jlzdvhrfrc2nexwptqyzcwufs4pqy2qtegy,false,"<A1>epcids<81><D8>*X#^@^R z
<95><A5>Y*<AC>G5^N<80>[    亽^V<F0><FF><EE>$<C7>^W^@zX<8C>""|}<E5>֝"

Hex equivalent of the label as it appears in CSV:

a165706369647381d82a58230012207a95a5592aac47350e805b09e4babd16f0ffee24c717007a588c22227c7de5d69d22
iand commented 2 years ago

The first of the unusual labels was initiated by message bafy2bzaceb6vyb62qkicnoz2jdrcdtkbb6xt5pvc2ygbxn36qrcm7hsbbggzi which parsed_messages encodes correctly:

95384,bafy2bzaceb6vyb62qkicnoz2jdrcdtkbb6xt5pvc2ygbxn36qrcm7hsbbggzi,f3u6nbhlifh23a7qhl4qnggdmr6gxqqugcgegvldj5z4uuqcwgzg7pggjr7fpvlshg5ytegdyheqgs65ohghpa,f05,0,PublishStorageDeals,"{
    ""Deals"": [
        {
            ""Proposal"": {
                ""PieceCID"": {
                    ""/"": ""baga6ea4seaqnxgztgkfpklt4a2m7jlzdvhrfrc2nexwptqyzcwufs4pqy2qtegy""
                },
                ""PieceSize"": 262144,
                ""VerifiedDeal"": false,
                ""Client"": ""f3qv4q242k5zmdbpztrndzymnls5z3lomvy7ocmizd7yfrgvvbve7se5of4uygmkibg42j6m7r46xptzbgbngq"",
                ""Provider"": ""f09833"",
                ""Label"": ""\ufffdepcids\ufffd\ufffd*X#\u0000\u0012 z\ufffd\ufffdY*\ufffdG5\u000e\ufffd[\t亽\u0016\ufffd\ufffd\ufffd$\ufffd\u0017\u0000zX\ufffd\""|}\ufffd֝"",
                ""StartEpoch"": 98260,
                ""EndEpoch"": 1139144,
                ""StoragePricePerEpoch"": ""122070"",
                ""ProviderCollateral"": ""0"",
                ""ClientCollateral"": ""0""
            },
--
    ""Expiration"": 1647714,
    ""ReplaceCapacity"": false,
    ""ReplaceSectorDeadline"": 0,
    ""ReplaceSectorPartition"": 0,
    ""ReplaceSectorNumber"": 0
}