the pds just deleted all but 4 of my blobs

[goeo-]

Describe the bug

all but 4 of my blobs just disappeared, with no explanation as to why.

Expected behavior

my blobs to continue being hosted by my pds

Details

• Operating system: arch linux
• Node version: using the docker image ghcr.io/bluesky-social/pds:0.4, original compose with just caddy removed

[bnewbold]

That seems concerning!

Can you give more context? How many blobs were there before; what does the listBlobs endpoint say about the blobs; did you delete any records at the same time? Blobs generally only stick around if they are referenced from records in the same account: uploaded blobs get removed after a short period, and when records are deleted, referenced records will usually be removed (unless another record in the same repo still references that blob).

[goeo-]

A lot more than 4 on listBlobs: https://goeo.lol/xrpc/com.atproto.sync.listBlobs?did=did:web:genco.me

I didn’t delete anything, no clue what triggered this.

Example missing blob: https://goeo.lol/xrpc/com.atproto.sync.getBlob?did=did:web:genco.me&cid=bafkreihuotd4ehd6tab6ztq4k76hcmrllydakjrgoyw5v3paorvu6imoki, used on https://bsky.app/profile/genco.me/post/3knsimmbj4c2x

[goeo-]

Oh, more significantly: https://goeo.lol/xrpc/com.atproto.sync.listBlobs?did=did:plc:f5ejbog3ostso47er2y5ugbl

ALL of this account's blobs are gone.

[whyrusleeping]

I'm not the right person to help with the bug itself, but I have 36 blobs cached from genco.me, and 58 blobs cached from did:plc:f5ejbog3ostso47er2y5ugbl, likely can find more of them around in the CDN too.

[dholms]

Definitely concerning and surprising to hear!

Are you using the disk store or did you set up s3? Did you use the installer script or handroll this yourself? Are the 4 blobs that stuck around newer, older, or a random assortment?

One possibility is if the directory isn't correctly mounted in docker, the blobs may not have been persisted after the container restarted.

It'd be helpful to find out if the blobs are still stored on disk / stored in your account DB. If you're up for running a few commands you can:

• check if blobs are on disk through ls /pds/blocks/${DID}
• check if blobs are still indexed in your actor store by
  • locating the actor store find /pds/actors -name ${DID}
  • querying the actor store echo "select * from blob;" | sqlite3 /path/to/actor/store.sqlite

[goeo-]

Hey,

Are you using the disk store or did you set up s3?

disk store

Did you use the installer script or handroll this yourself?

hand rolled but following the installer script, i used the original compose with just the caddy removed

Are the 4 blobs that stuck around newer, older, or a random assortment?

-rw-r--r-- 1 root root  46K Mar 29 21:26 bafkreibhcpwzw7ncbacr42w5fkald67swyvuyggjrvmo7b6v4lfhxoctfi
-rw-r--r-- 1 root root  78K Mar 29 00:53 bafkreiewosnifxc6big5efps6la5gp44n6ueyjkwde3nllrjsfxticnyp4
-rw-r--r-- 1 root root 113K Mar 28 23:13 bafkreie7uta5ijlfdzhfdp75k65i4iaksial4rnj7rfddrampcygzfligm
-rw-r--r-- 1 root root 222K Mar 27 15:27 bafkreigyohqfm5ukh7dmmnolsluuo2irfxrgra2ozbsvwxis2gig4bx4o4

I believe these were the remaining ones when I noticed this, which do appear to be the newest 4 - so the issue could be that it deleted everything and I noticed late, actually.

One possibility is if the directory isn't correctly mounted in docker, the blobs may not have been persisted after the container restarted.

hm not sure why this would happen but, say, my signing key would remain, when /pds as a whole is mounted.

check if blobs are on disk through ls /pds/blocks/${DID}

:) find blocks     
blocks
blocks/did:web:genco.me
blocks/did:web:genco.me/bafkreigyohqfm5ukh7dmmnolsluuo2irfxrgra2ozbsvwxis2gig4bx4o4
blocks/did:web:genco.me/bafkreicbt4udliyd3pmu65nj34xyh3jzxt46qwchi7tl7xw6l3fjytllem
blocks/did:web:genco.me/bafkreiewosnifxc6big5efps6la5gp44n6ueyjkwde3nllrjsfxticnyp4
blocks/did:web:genco.me/bafkreicvhinuzbq3acyg4rp6owclzy7bv2cnamf73rgdsrgfdtujyjfv4m
blocks/did:web:genco.me/bafkreid7wway5e3ge33tlg7focsbfdgwqo3lmkok2ioxrlcqyglbmj4hz4
blocks/did:web:genco.me/bafkreibhcpwzw7ncbacr42w5fkald67swyvuyggjrvmo7b6v4lfhxoctfi
blocks/did:web:genco.me/bafkreifil2zw2r6knohsjaup2dhdjz2ecok4es45rkkb3oy7xinyjhh6wa
blocks/did:web:genco.me/bafkreie7uta5ijlfdzhfdp75k65i4iaksial4rnj7rfddrampcygzfligm
blocks/did:web:genco.me/bafkreia5blvaztukfz555aqiy6twvhdzx2n57wp53vfubk6vduvgfx6cv4
blocks/tempt
blocks/tempt/did:web:genco.me

check if blobs are still indexed in your actor store by

i believe this was also accessible through the listBlobs calls above, but:

/pds/actors/45/did:web:genco.me/store.sqlite
bafkreiacosy4nqvvqic6iel5yr4srr3guykk36lyruu4j2hejv7s4uehuy|image/jpeg|131017||860|564|2024-02-23T13:55:03.575Z|
bafkreibo3lyfwt4ygcyah3rvrzml6dbdiqwmhgt3v554pesqxc4uzic7kq|image/jpeg|642613||1640|1240|2024-02-24T20:55:57.387Z|
bafkreih5btrtaqb7usmyihnmc2io7ynibg2i24fbzfzqbp66aaqjnrvgzm|image/jpeg|15836||398|70|2024-02-28T07:47:47.148Z|
bafkreiarelgr3t7dstqakaz5cqocrd2ypahxeiucpjrm2eylhbcs5hvfee|image/jpeg|359096||2000|1484|2024-02-29T02:40:51.878Z|
bafkreigdtem2lvac3o4erc4b5rz3kj3monjtqi6hrmct4hfrwp57cjur2a|image/jpeg|340841||2000|1495|2024-02-29T02:40:52.533Z|
bafkreid3ccnrbut5cet5vissallfvkaapaz6s3335ibpsrxuinseo35dri|image/jpeg|311373||2000|1000|2024-02-29T09:36:44.615Z|
bafkreic3g3wnwk6m4wgbpwr5upttbam7vybe33hsaidybtxfzok5xwq6ri|image/jpeg|667276||2000|2000|2024-03-01T12:14:54.320Z|
bafkreifi5nve3jwwic7r77h5vmfwbaxc5223vefbsflerp6ufhq3w5jcfy|image/jpeg|166549||2000|541|2024-03-01T14:27:18.358Z|
bafkreibje2n6qeeduvypr4hui7jdg4w6eabdjbhraespepxcaa5icsgflu|image/jpeg|325395||2000|1214|2024-03-01T14:27:55.295Z|
bafkreibulgf6w5ilbma7oma5c6mux64thsodslzhwbb2r6x6kdcacdpvka|image/jpeg|441363||2000|230|2024-03-01T15:21:46.634Z|
bafkreiccjpk3giryyeqdamjtyyd7f46kd3b4smol7jyaevenhczfuv25le|image/png|164478||1062|796|2024-03-01T15:24:58.756Z|
bafkreievkzgbsupc6k24vcmy5dokk6cc4kp7btqn3f3fnndop7hnwrm7me|image/jpeg|834579||2000|311|2024-03-01T15:44:09.218Z|
bafkreiequ6jvcggz5qlt5dhiadzvdke3xsojfni3y2y2s4ekekbibm2ocq|image/png|469921||3420|268|2024-03-01T15:47:33.341Z|
bafkreib7yhs4stmcmmivno5qicmmy4kb4xavo5ewbb3amiuk7pbtpomvyi|image/png|851214||1676|1774|2024-03-01T15:56:22.057Z|
bafkreidt3n7ed7ytukdhw2wyrwced4mnz7loffloyqdm7ojt7qk47naz6m|image/png|30536||772|360|2024-03-01T15:59:16.246Z|
bafkreigwqv3g6bfsyjlqdt7xyp3nxj3zb3qwqb3u23g7iivew5wvpkbdpq|image/png|82840||788|822|2024-03-01T16:12:28.704Z|
bafkreidxis76fvwkbmfrka5cebcwukusgfjfvwkd6pxvrsjcevaw3k7zyu|image/jpeg|959381||1170|1907|2024-03-01T17:25:08.500Z|
bafkreiguznbpsm4ldipspv6ndulqrrghv73r4oyfzdmx6q5xsetvdu3way|image/jpeg|607677||1170|1171|2024-03-01T17:25:08.943Z|
bafkreidclt6iiobgu4fscj6bahaug26bigunzzs4ylrakmzx6rxaubwoze|image/jpeg|494358||2000|1142|2024-03-01T18:29:13.050Z|
bafkreich45jiod5d4mjce457xiajrvtbbpe2kivc367zcqf4ftojjjb7ey|image/jpeg|367853||2000|1000|2024-03-01T19:30:54.322Z|
bafkreiglriszww7wtdi2bt4kwrbwnwiulds3hihjz3sak2e42ut55mkxrq|image/jpeg|472114||1170|1156|2024-03-02T21:13:48.077Z|
bafkreidcjx2trc2gvwxfoo6rc33ewfddmqkyo7e6uvcz4s5pk6e7run3te|image/png|70915||1200|600|2024-03-03T03:59:18.300Z|
bafkreid2mpbqbk2i6ab5znu2xle7jygnyglcxrixzeajgm2nvu5bhpce3i|image/png|82359||1200|600|2024-03-03T04:12:25.438Z|
bafkreigjp75ffizi5mzzz4ue5hzmfaioz4peoxloklmkvhj5r6qq4viudm|image/jpeg|298563||2000|1000|2024-03-09T03:17:48.021Z|
bafkreic3hteibiplva527bccnfw6jij6jet4mgahyjkh4bcmgmr76yvwbq|image/jpeg|563179||1920|1080|2024-03-12T08:29:32.285Z|
bafkreif25oizf3byenzx4sizri74ji6whzujfyl2udts3fco3wxdnjq7a4|image/jpeg|589532||936|936|2024-03-12T08:39:27.723Z|
bafkreibt2vr62iqza3lmrqdpggm6vayvlxk24grgz2j7c75yi2hrffwjtm|image/jpeg|560300||1640|1905|2024-03-12T12:57:04.721Z|
bafkreihiczwhj4uewakshp6eq4n46qgnnsbov2gueibln436xezg63bwwi|image/jpeg|165908||2000|1000|2024-03-13T05:31:36.221Z|
bafkreidocgvumw6n2wrhzexnoy5tpft67uf5inxp5g3hpiluu6757ydbhe|image/jpeg|590751||2000|1500|2024-03-14T18:44:28.108Z|
bafkreiccdwy6fbe5xlx3ymmizwnqdhsbpjfm4lb6wdujxbu243ex3oaxiy|image/jpeg|189026||786|768|2024-03-16T03:19:18.689Z|
bafkreib7uwp577l77r7iqb2vhbryo7jolzgl2jubwhyso4aojr24xn34iq|image/jpeg|421628||1170|1568|2024-03-16T03:50:42.130Z|
bafkreifp6f6kytogzmdxqbq5yeyvaz3qxbyzv5jlpn53oz2pqzi3swv4nq|image/jpeg|320747||2000|1000|2024-03-16T05:51:59.628Z|
bafkreifwuvejtc4k7aguo3z6u5q5pihjzgy5mvvova4dimrsq3l3kqe2ja|image/jpeg|175777||1170|520|2024-03-16T07:39:23.590Z|
bafkreib6mycuxngjpc4h5gk46s2r5zotnqmtqqnkkg4n5sns2h6pcjje7m|image/jpeg|83717||976|204|2024-03-16T08:53:40.119Z|
bafkreihuotd4ehd6tab6ztq4k76hcmrllydakjrgoyw5v3paorvu6imoki|image/jpeg|907302||1080|1124|2024-03-16T09:21:09.344Z|
bafkreig66ezwsxe7qfhrfkflchy2ik54idmbvjh3yp4qrkztgztbuasaxa|image/jpeg|555886||1118|889|2024-03-17T22:36:35.628Z|
bafkreiacgw6by3v6irpww2faqzoxki2nlbsjre4crqofrwny47iy7uvffq|image/jpeg|241910||2000|1000|2024-03-20T10:24:27.076Z|
bafkreifsy5xt5c4kfj7z7b7cxd5ojmkxsw4fvmy7q3pdpgiljszhs2xb2q|image/jpeg|346127||2000|1000|2024-03-20T19:21:44.412Z|
bafkreiebkegsrb4lfmg6pstqtnnq42egkmhqvtgstz3tsfambmlwfthvnm|image/jpeg|57849||702|112|2024-03-21T17:07:03.584Z|
bafkreifeqnujjucok3dsq24keajapyutir73bc3stnz73zhzcf4jltgi5u|image/jpeg|92130||1054|110|2024-03-21T17:07:03.730Z|
bafkreid73toscmpfybb7trwejbjdmchj3xuxvbiphees7cxpkvz2v4dgja|image/jpeg|76175||818|88|2024-03-21T17:17:45.216Z|
bafkreicxexgwqp7zypfvqgaukcn5vaz3khqck25e5x7cfqohacuqq4z4ty|image/jpeg|354848||856|1084|2024-03-21T19:01:19.062Z|
bafkreicq5eyyh6hqtn2ip2kd3jcdqdg74slhqvawntl75vu3d3hdtpxmne|image/jpeg|220673||458|1082|2024-03-21T19:01:20.128Z|
bafkreibjgsa4fxk7inpfh6cl554azet2hrqre55eoghgdl5tpgowfk6vpm|image/jpeg|801968||2000|1500|2024-03-21T23:42:13.128Z|
bafkreigyw2ytpk73um3ibise662taypzdwquxnexzbe2kkgc7nrteudwpq|image/jpeg|184018||1034|396|2024-03-22T08:34:12.402Z|
bafkreigrhymp2hmwj456uhs4ukjtc4loaw32fc2xsy4dq2m6tl7xtckiiu|image/jpeg|121570||2000|1000|2024-03-22T09:45:00.961Z|
bafkreieaa6vo434c5aphlwu6c3wpswoetehrhfvexmicceiydohqygqbhq|image/jpeg|243315||2000|1125|2024-03-22T10:47:28.235Z|
bafkreiauqk7hnrgtar5f2q7ojgh7atq5rbbrtag7nnk3akfkclkqj2o5zu|image/jpeg|34031||180|344|2024-03-22T11:45:22.697Z|
bafkreiabk7hydvxmea3uz5hx46mvtqsb22s5nsrmebhy56jah7td264674|image/jpeg|12819||180|180|2024-03-22T12:12:58.541Z|
bafkreie245q2sgv5u2f5j6wy2dhnreuikyh3mqvu2u7nylgl4xf7p5pl34|image/jpeg|109717||904|664|2024-03-22T18:53:06.119Z|
bafkreidsimnfywroelco4wiylenjzdmb4id3u5grj3i6jwanfisni6eewq|image/jpeg|61783||834|242|2024-03-22T21:26:32.590Z|
bafkreibwl46d5cmfwhxgxk7epye5xvmx44vljn73s5mpzmo2lvveilwjju|image/jpeg|61261||712|302|2024-03-22T21:28:02.721Z|
bafkreibxfx5idkbsk4kum5ur2mfdq2ypakcjvqddkms6qvodwidkxtoho4|image/jpeg|69803||710|278|2024-03-22T21:34:54.235Z|
bafkreifr2beu5gpnzutlmtsdarlh7nthd2qml7p6tfa6h6iutbfooofowy|image/jpeg|154545||1500|1212|2024-03-22T21:52:30.797Z|
bafkreie3liu2z4yhcmsd5jxcf2jngecgdnxhbzyczohumkgnr3fciqk4ii|image/jpeg|184772||1212|688|2024-03-22T22:12:58.519Z|
bafkreic2n2re2gim523jlokygz5yjfk6wbf36xy3an3c52ojh7dr7flace|image/jpeg|11542||364|112|2024-03-22T22:26:33.035Z|
bafkreicalhaqhsr5expqi23yo67p7szlswqfte2ygm3pt74dchol4uqt4i|image/jpeg|592689||1562|714|2024-03-23T00:25:59.205Z|
bafkreic2ycsxcy6iufvx2vs7mmp3dzc7tfwfd4b7hikgn62ctlvtass4l4|image/jpeg|121447||1404|568|2024-03-23T00:25:59.558Z|
bafkreidtknw45nmyonmnvmqokp4iyz3el5wy3zkzifqnbxub3z6vdhzyji|image/jpeg|119782||1170|409|2024-03-23T00:39:56.947Z|
bafkreidk2rzvjc2yjgwurwqzlmdltm6qsb4hicjpq6w64szqvbsyq6cyam|image/jpeg|369585||2000|1000|2024-03-24T06:51:41.235Z|
bafkreial422vl7ikdlqb67sgobvumcpyn4n6fy2jcv2axk4vhu37ywkhs4|image/jpeg|685132||940|720|2024-03-24T13:20:25.874Z|
bafkreiaz4c4ab3gqaa57vismlru3cw7muys5anzdvrrdre6ivvu4qazcee|image/jpeg|45862||1160|140|2024-03-24T15:42:46.045Z|
bafkreieez2c77wxh2bscecmagsbjshhatnxp6ah7emdz3hntxe54jssazq|image/jpeg|430650||1490|742|2024-03-25T10:26:35.069Z|
bafkreiaym6ghcenfg22mjwfwibzqhr43ssb6ugmo3b4z7c23yfwmc3idwq|image/jpeg|716539||1236|1582|2024-03-26T18:14:47.166Z|
bafkreig7v2agkh3tathgek2celvs2fnuet22zsegdkovagpumwkdk7y6xa|image/jpeg|135958||1192|390|2024-03-26T19:03:49.821Z|
bafkreiamxcgpuxgt6xqmanq45a5ladwus6ouvutvifwu52wunewxgw7v5a|image/jpeg|39268||980|112|2024-03-26T19:13:41.716Z|
bafkreigyohqfm5ukh7dmmnolsluuo2irfxrgra2ozbsvwxis2gig4bx4o4|image/jpeg|226884||1170|786|2024-03-27T14:27:38.206Z|
bafkreie7uta5ijlfdzhfdp75k65i4iaksial4rnj7rfddrampcygzfligm|image/jpeg|114948||866|472|2024-03-28T22:13:43.054Z|
bafkreiewosnifxc6big5efps6la5gp44n6ueyjkwde3nllrjsfxticnyp4|image/jpeg|79681||1010|258|2024-03-28T23:53:51.189Z|
bafkreibhcpwzw7ncbacr42w5fkald67swyvuyggjrvmo7b6v4lfhxoctfi|image/jpeg|46770||488|262|2024-03-29T20:26:22.297Z|
bafkreia5blvaztukfz555aqiy6twvhdzx2n57wp53vfubk6vduvgfx6cv4|image/jpeg|298017||2000|1000|2024-03-29T21:22:38.244Z|
bafkreid7wway5e3ge33tlg7focsbfdgwqo3lmkok2ioxrlcqyglbmj4hz4|image/jpeg|259952||2000|1000|2024-03-29T22:06:20.485Z|
bafkreicvhinuzbq3acyg4rp6owclzy7bv2cnamf73rgdsrgfdtujyjfv4m|image/jpeg|58432||1000|1000|2024-03-29T23:48:05.711Z|
bafkreifil2zw2r6knohsjaup2dhdjz2ecok4es45rkkb3oy7xinyjhh6wa|image/jpeg|290856||2000|1125|2024-03-30T22:19:27.268Z|
bafkreicbt4udliyd3pmu65nj34xyh3jzxt46qwchi7tl7xw6l3fjytllem|image/jpeg|260357||2000|1000|2024-03-30T22:48:26.240Z|
/pds/actors/35/did:plc:f5ejbog3ostso47er2y5ugbl/store.sqlite
bafkreiaa7qk6epmity4yqzwnlp2ntxhkelz5y3zvzjersbzpujqve7nyke|image/jpeg|344333||640|853|2024-02-22T22:31:57.895Z|
bafkreiahgtkvrohj63byu2hmw4cdkejci5gqnfqtxswto36mrvuxtvwj5m|image/jpeg|108783||752|368|2024-02-22T22:31:58.225Z|
bafkreiakvztgt45gyb66gth2tz7tl4xq7pxhp4dowfkusipsmq3y2hi3ki|image/jpeg|794578||2000|1125|2024-02-22T22:31:59.521Z|
bafkreianuqkewh2hb4mm56fzh4ts4be4hnuqc4dmlbbiar2zyqrz2v7yyy|image/jpeg|296483||2000|2000|2024-02-22T22:32:00.305Z|
bafkreias7jejg7jrondboqefqep5qv6khxxsjq6jqgvwwpmb3its4dbug4|image/jpeg|724586||1896|1896|2024-02-22T22:32:00.903Z|
bafkreiauhgng26pizjabf77sfavwf52opnfm5fvgzzd3ra2jda3rn36fiu|image/jpeg|661963||2000|1281|2024-02-22T22:32:01.585Z|
bafkreib22y6q37l5533mlsuevieiqicq6ydpizxxgawvtr6qzotcmicici|image/jpeg|191691||946|404|2024-02-22T22:32:02.435Z|
bafkreib4turnffiyki5m2utc4ktjvvxhccmve7hbjgxwzf27wnmvhaasnu|image/jpeg|366263||1170|866|2024-02-22T22:32:03.015Z|
bafkreibkmp4rqzykx66ryou5sbwcitg2kzywr4g23rcaoh5lckepshpxr4|image/jpeg|149498||2000|1000|2024-02-22T22:32:03.383Z|
bafkreibkzrl6umqtjzj3st2alerazxqljqj53xwgi6z5dat5fn7lwcn3f4|image/jpeg|929330||1826|1420|2024-02-22T22:32:04.099Z|
bafkreibnb76nxak6w3fnmgkack22moib5gkmc7pmwpfsaj23cjpxovaeey|image/jpeg|44172||582|318|2024-02-22T22:32:04.419Z|
bafkreibndcvafjp55qcvwypakj4ey4vm3frex2stg2nitnuogg4am5iejm|image/jpeg|44147||430|138|2024-02-22T22:32:04.735Z|
bafkreibp42mocaqfum7o6ehp3p2d4oqm3rrv46oojrhmmlkhe33rja3icu|image/jpeg|404715||1090|1236|2024-02-22T22:32:05.464Z|
bafkreibqp57hdonqcmtykrudrdttytuvbqq7zy6sn7pnqd265aotnydh7e|image/jpeg|857277||1171|1560|2024-02-22T22:32:06.346Z|
bafkreic2egpxhrasdpzslvk7giq2zipj63xwwqrtqzxvtg4zook3y3ldv4|image/jpeg|362786||768|816|2024-02-22T22:32:06.749Z|
bafkreic4m65usl5a2jcnprkc4rsaiyp5i4q23lxm33gshgnn2dunaeinoe|image/jpeg|371729||900|601|2024-02-22T22:32:07.209Z|
bafkreic6uybrtfp7ekjtxmkjaq6ah6i43g67rxh7rcvutcnnocyabtievm|image/jpeg|839535||2000|2667|2024-02-22T22:32:08.140Z|
bafkreic7ijynowkalqsrkdzplkc4l7acvhrosoowtfijnf34oxfcp7rxjm|image/jpeg|323199||1170|804|2024-02-22T22:32:08.804Z|
bafkreiccmqg7tljadojkc55lxoe73vwhiazx7v5ibp6lpj24g3rilaxriy|image/jpeg|18965||480|68|2024-02-22T22:32:09.224Z|
bafkreicd2wpxoxkm5uciwo2oidyferdwrtufzpm6vjxkmvffxitgtl2aiy|image/jpeg|221940||1170|672|2024-02-22T22:32:09.814Z|
bafkreicf43fnl3n5twpkw23tlejcksomxivxnyrkuigw2nhunezhduzeem|image/jpeg|469419||2000|1107|2024-02-22T22:32:10.511Z|
bafkreicgoomtcaib2ydrmoeobpeide3wvm4n4mydr6mzkd2muvbxmdj6sa|image/jpeg|968210||3556|2000|2024-02-22T22:32:11.237Z|
bafkreicqn7lzazubuvg566i3chmapouq7udvxdtoohmondlfgjoa7khy2e|image/jpeg|262345||734|434|2024-02-22T22:32:11.847Z|
bafkreicuir3cfulq2cyp5ho5sr37y4drshfbqwauekol5hprrrzxhu4vyy|image/jpeg|45750||594|224|2024-02-22T22:32:12.530Z|
bafkreicxjm6ceisd6qgehhamwsc4xymhyjyrcv4dwyc2gdsqhzw6wrm4hy|image/jpeg|7153||226|52|2024-02-22T22:32:12.921Z|
bafkreicya45fzi6pbcsy76cr67632pdenwhtxxitbnvk3qzhp7s4ewfunq|image/jpeg|288147||900|601|2024-02-22T22:32:13.721Z|
bafkreid2k7crdxcy3hhilrm2htjjtf4knnhrn6m2iyb25us52mngytct2a|image/jpeg|187075||1170|896|2024-02-22T22:32:14.256Z|
bafkreid3ccnrbut5cet5vissallfvkaapaz6s3335ibpsrxuinseo35dri|image/jpeg|311373||2000|1000|2024-02-22T22:32:15.231Z|
bafkreid6tyxgdm2ad2gq43hnyxkjmbtwnpkmorjcfyc6ownzvzm5d6mty4|image/jpeg|102841||1244|224|2024-02-22T22:32:15.593Z|
bafkreidazntz27fnbpz774qjtmnjm3446imrjplneb323wlpaeodxxpgha|image/jpeg|727696||1774|516|2024-02-22T22:32:16.547Z|
bafkreidfhqc5c6iuxeosrbodrhuwetjqfzpv3vqbshasp7swtk7v3chkjy|image/jpeg|111905||1170|370|2024-02-22T22:32:16.917Z|
bafkreidgxuqlq5ads3ddkibzdgxeavwdvehfgvmayhtlop6rxxlbw2xmlq|image/jpeg|181760||538|404|2024-02-22T22:32:17.327Z|
bafkreidhhkqorjsse2jiuolqo2f3kmqg4h2ljztr2xwz4ofa2z5t3tyhz4|image/jpeg|188432||1842|246|2024-02-22T22:32:17.852Z|
bafkreidk7f36gaxnug4fkkmae7dlam7havzj2xgzf6krgexd42rafyvtj4|image/jpeg|75333||864|348|2024-02-22T22:32:18.230Z|
bafkreidlswo6vn6kkrhfmy5umgambe2q7sqprxksq5unygw4fzbhko7oxq|image/jpeg|200081||1170|516|2024-02-22T22:32:18.728Z|
bafkreidtyn44oepxwlavhwwbb55knsbm34ok2mrszafewdniyak6lnqo44|image/jpeg|399242||628|626|2024-02-22T22:32:19.747Z|
bafkreie5yeqhfabji7yq57a56k5ddmgc3drrhbl4bfuondt7js7dvz6s2a|image/jpeg|242401||1080|752|2024-02-22T22:32:20.516Z|
bafkreieb327wjqiyc2qy6j2gcj3byum3zavbnstdo25yyqy3alvptjl2xi|image/jpeg|161270||1170|332|2024-02-22T22:32:21.083Z|
bafkreiebexsnr4rxdtexcatxqx5wfwsnnkipnqlvxtieluzpzpyqxepoum|image/jpeg|660142||925|2000|2024-02-22T22:32:22.037Z|
bafkreiedtgljplhibkj6h6b5u7p5f7t5kwxr5znwhtx23lamnoxvpbbplq|image/jpeg|92652||396|484|2024-02-22T22:32:22.446Z|
bafkreieiapmto7xfvfn35ce5v2snrwdndpsp7pfcht3rrkdlbnv6so6h5a|image/jpeg|20060||746|68|2024-02-22T22:32:22.817Z|
bafkreieknhu6ytetxcsft2sdptzqq66yj73vqu4famgmmdsnhqbki3ej7q|image/jpeg|758494||925|2000|2024-02-22T22:32:24.779Z|
bafkreiendv3dfh42kgz47d4qf5r25dopnywsaywrbmi52436pq6mktonbm|image/jpeg|101269||2000|1600|2024-02-22T22:32:25.187Z|
bafkreienrhxa2qzbtavsuhidg44bhl4uv6mr7xaihlrbdmevfrmnri5llq|image/jpeg|32745||384|342|2024-02-22T22:32:25.615Z|
bafkreiernb5icq3ruje337yseovnik2onfr6zdqtpreb4oisx3ghzd2zra|image/jpeg|550415||2000|3158|2024-02-22T22:32:26.572Z|
bafkreiewmuehdbk4ewmmru6tox5hbkbgmzhphvfa73f7brxdxnvzadjxpa|image/jpeg|808776||2000|1125|2024-02-22T22:32:27.533Z|
bafkreiexehos3f6rs37b2mfi2ahejtg3kloac3q64ud3qbqgdbnmrsm5ku|image/jpeg|179486||1170|416|2024-02-22T22:32:28.821Z|
bafkreif3ys746d2mcdpfdl3i2v2sfcjzcpzmp7wlbfh4qsiwvqpnwevjre|image/jpeg|742017||925|2000|2024-02-22T22:32:30.615Z|
bafkreif6xbheauhldjiudrrrntjc3vrp4yakwffo4eduv7q2boj4p3ov6a|image/jpeg|123576||888|394|2024-02-22T22:32:30.998Z|
bafkreifbeifzcp2qt6nrfwjx776it5cgm5xgac4lrbbt3yvjtqy7zvbv2e|image/jpeg|420281||2404|2000|2024-02-22T22:32:31.766Z|
bafkreiffinq5pzkhmxuuqvtjh7q7i2pdvvr7vszlfkr6zesyhpi5rjrhva|image/jpeg|104043||780|170|2024-02-22T22:32:32.162Z|
bafkreifgtirymti43awzx6qngwxxo5sbqljfk572zfdtiku547bk4p6rvu|image/jpeg|306974||2000|1125|2024-02-22T22:32:32.733Z|
bafkreifmkub7notnseoeg2qyqddbql225cuc2njc3d75bopo5tx3dkaiou|image/jpeg|256272||1054|656|2024-02-22T22:32:33.412Z|
bafkreifpamwgi4ef46fjg3aldaoptbnbd7dk44e5tzeb7yqfmf4jlvzv6a|image/jpeg|353590||925|2000|2024-02-22T22:32:35.238Z|
bafkreifposu6zekgeoeoqq5a7pj4bzt4sbe6ezpgibmoxjsaurhx4m2dsm|image/jpeg|97276||1012|186|2024-02-22T22:32:35.669Z|
bafkreifq7pii5lg3lsvqppwwsy6itivngjnopyqslfge5drhijkuuqlqma|image/jpeg|149072||1072|420|2024-02-22T22:32:36.153Z|
bafkreifqbhum5kyd3ricahljmxtrbi7hanwau5ayuxezub2woiqud4vrii|image/jpeg|855950||2000|2000|2024-02-22T22:32:37.109Z|
bafkreifviqzvey74s4pcgemha37ld5tszl4zzt3yql4l3rdj46bpasr5zi|image/jpeg|426731||1551|523|2024-02-22T22:32:37.882Z|
bafkreifxxbaxffc3d5ifn7j7uxsco7zlnpopkxpa5n2psfjpgess3vh2sm|image/jpeg|791431||2000|1125|2024-02-22T22:32:38.714Z|
bafkreifzw5vshlkm3zvulyndbdwvomwqdrmahpxg6owxokech333z4zbie|image/jpeg|217899||1350|618|2024-02-22T22:32:39.422Z|
bafkreig2aawndulvppfoekrpra5ge6vnoj6hz5xirv4ybzbdrfplev7ggm|image/jpeg|104386||956|318|2024-02-22T22:32:39.832Z|
bafkreig4dfksuhb6r2mu4fh6ja4c2uzjcyv35uuair76mx63aswaukihjq|image/jpeg|83572||2000|45|2024-02-22T22:32:40.829Z|
bafkreigc3n5e3pzbulm6c4f3dtpfdmkbfnbwkeb3q5r6zlcvquilyg3yry|image/jpeg|683467||2000|1125|2024-02-22T22:32:42.060Z|
bafkreighb5nk4d4hk3cwgtoymfx7mwop5xr3enucqnqqntqjqtp4xwmh5m|image/jpeg|119454||790|538|2024-02-22T22:32:44.619Z|
bafkreighprejgppbz27fi72u3f4zfjksknzkrcxzp5dyit7aklijqfnsom|image/jpeg|255117||1170|726|2024-02-22T22:32:45.379Z|
bafkreigtnclsds4cjoaxl2lvc5zuruo6yoqphdcqryl4efi27lajl2mgu4|image/jpeg|415344||1170|1408|2024-02-22T22:32:46.166Z|
bafkreigyo7ioo6i7oxgxb3xwdewjbcgmt6h7zjfnxb5nooilsfvqtoob34|image/jpeg|223367||756|402|2024-02-22T22:32:46.587Z|
bafkreih6dzhu6evnvc4zx3v3nqt7r7jm4mxuguo5xyti67bg2b4fe475xm|image/jpeg|632194||925|2000|2024-02-22T22:32:47.677Z|
bafkreih7x36bbs46xeuqqark2ruchj6np5pkxmpyzeglrrr4zypsqtdvne|image/jpeg|82542||928|354|2024-02-22T22:32:48.078Z|
bafkreihbv2gzw7z2duesitjaznmwxrvhns3vh7jjihojrg5wo4qre3f5iy|image/jpeg|150271||954|241|2024-02-22T22:32:48.568Z|
bafkreihbz7gk4xfio6jm32gwvjmqeru363aqxxhd55wk6pr6mp3uvmax7a|image/jpeg|55410||630|268|2024-02-22T22:32:48.915Z|
bafkreihd5ws4f2vo7vcjuai6j4wquddsaf6mb6dunq3onusiayoiyzhprm|image/jpeg|210991||2000|2000|2024-02-22T22:32:49.782Z|
bafkreihdb7kddgzmaenldhyklfhtz622widpsnhqmfiwlfhaud4av62poq|image/jpeg|7153||240|56|2024-02-22T22:32:50.117Z|
bafkreihdekhk7eyq5ccf7wlnbazsr47gjdlxw7eqzfnui4qdp2gxgwszd4|image/jpeg|42133||432|98|2024-02-22T22:32:50.480Z|
bafkreiheqdg74udnnnglmmc2x7iwqhsu7tkqhbxhen4wouwm6b3nx5aljm|image/jpeg|422613||1454|1896|2024-02-22T22:32:51.570Z|
bafkreihetfig3buu5afcxcm5oe2nphweikmviue77zldhakr3iygw5iqom|image/jpeg|845846||1170|1626|2024-02-22T22:32:52.725Z|
bafkreiheuo63dxchxyakgeawhpeqrwaib3g6mdqc4nhkg666huop2k44lu|image/jpeg|19308||316|90|2024-02-22T22:32:53.070Z|
bafkreihgkq3ubd3xcqxzzgesdv75oeefu3by3jgjldhy5qstue46c2jlra|image/jpeg|787788||2653|2000|2024-02-22T22:32:54.549Z|
bafkreihgqywl3wnw2pwxaxsy2nk2wro4omvzgptuhds7fvobjtcmgk766u|image/jpeg|30676||474|90|2024-02-22T22:32:54.908Z|
bafkreihownjf2yh6zctbany3km5tejc4rqrwr7wccpjc42kk6y7lnkwste|image/jpeg|625911||1470|1210|2024-02-22T22:32:56.323Z|
bafkreihqo3qfaypj7w5lr7pz6lgbcqmc76rjjfe43zsjtrsw4olqy4bhaa|image/jpeg|279510||1170|828|2024-02-22T22:32:57.110Z|
bafkreihwk4s6wq3euauf3yyztnm2duzlsiacvrak3ssg3j7xc4d6qojbbq|image/jpeg|8394||254|60|2024-02-22T22:32:57.479Z|

[goeo-]

I'm not the right person to help with the bug itself, but I have 36 blobs cached from genco.me, and 58 blobs cached from did:plc:f5ejbog3ostso47er2y5ugbl, likely can find more of them around in the CDN too.

@whyrusleeping would be nice if i could have them! may be interesting to put some back in and see if they get deleted again, too

[whyrusleeping]

goeo.tgz

[dholms]

Did you ever change around your env file? Specifically PDS_BLOBSTORE_DISK_LOCATION. That would have caused your blobs to be written to another location.

It looks like everything in your blob table looks right - all blobs were linked to records and made permanent. There's no code path that winds up with the blobs table looking like that and the blob itself being deleted.

My best guess is this is something that happened at the system/environment level. Very hard to debug without access to the history of your machine or another reproduction :/

My best recommendation is to backup your machine (I have recurring backups running on my self-hosted PDS) and keep an eye out for a repro.

[goeo-]

CONTAINER ID   IMAGE                              COMMAND                  CREATED       STATUS                 PORTS     NAMES
b35d20ab4368   ghcr.io/bluesky-social/pds:0.4     "dumb-init -- node -…"   2 weeks ago   Up 2 weeks                       pds

It's been running for two weeks (since the update I think), I see nothing in dmesg, files usually don't just disappear..?

[goeo-]

One more important thing I just figured out: this seems to have happened after a com.atproto.server.deleteAccount call

[okpierre]

Maybe fixed by this https://github.com/bluesky-social/atproto/commit/9b78f06134188ef88baf9d9b9d613dc0c10607d1

[goeo-]

does that mean… this would happen every time to anyone using disk store..?

[dholms]

Yup this was fixed in https://github.com/bluesky-social/atproto/pull/2381

This was very helpful for diagnosing the issue:

One more important thing I just figured out: this seems to have happened after a com.atproto.server.deleteAccount call

---

does that mean… this would happen every time to anyone using disk store..?

Yes that's right. This would happen if you're using the disk store and delete an account on the PDS. We use the S3 blobstore in all of our production Bluesky PDSs which does not have this same issue.

Thank you for the bug report and apologies that it took us a while to find the issue!

This fix has been pushed out on @atproto/pds@0.4.12. If you're using watchtower, your pds should have been automatically updated. Otherwise please install the latest PDS image.