odd failure (bug?) when inserting a block in cmsweb-testbed/dbs/int/phys03

[belforte]

'timestamp': '2023-08-21 16:43:45.550851848 +0000 UTC

DBS Server error: [{'error': {'reason': 'DBSError Code:103 Description:DBS DB query error, e.g. mailformed SQL statement Function:dbs.GetID Message: Error: sql: no rows in result set', 'message': 'unable to find parent lfn /store/data/Run2018D/ParkingBPH2/MINIAOD/05May2019promptD-v1/260005/E4DF44F0-8206-C84C-9BE2-80FD55210F3E.root', 'function': 'dbs.bulkblocks.InsertBulkBlocksConcurrently', 'code': 101, 'stacktrace': '\ngoroutine 2109027 [running]:\ngithub.com/dmwm/dbs2go/dbs.Error({0xae4c20?, 0xc0000a07d0?}, 0x65, {0xc0004e0090, 0x87}, {0xa1ae29, 0x2b})\n\t/go/src/github.com/vkuznet/dbs2go/dbs/errors.go:185 +0x99\ngithub.com/dmwm/dbs2go/dbs.(*API).InsertBulkBlocksConcurrently(0xc00042e000)\n\t/go/src/github.com/vkuznet/dbs2go/dbs/bulkblocks2.go:509 +0x4525\ngithub.com/dmwm/dbs2go/web.DBSPostHandler({0xae81b0, 0xc00053a390}, 0xc00019e200, {0x9feab6, 0xa})\n\t/go/src/github.com/vkuznet/dbs2go/web/handlers.go:542 +0x1274\ngithub.com/dmwm/dbs2go/web.BulkBlocksHandler({0xae81b0?, 0xc00053a390?}, 0x454134?)\n\t/go/src/github.com/vkuznet/dbs2go/web/handlers.go:957 +0x45\nnet/http.HandlerFunc.ServeHTTP(0x410000c00042a101?, {0xae81b0?, 0xc00053a390?}, 0x0?)\n\t/usr/local/go/src/net/http/server.go:2109 +0x2f\ngithub.com/dmwm/dbs2go/web.limitMiddleware.func1({0xae81b0?, 0xc00053a390?}, 0x0?)\n\t/go/src/github.com/vkuznet/dbs2go/web/middlewares.go:111 +0x38\nnet/http.HandlerFunc.ServeHTTP(0x94c100?, {0xae81b0?, 0xc00053a390?}, 0x11?)\n\t/usr/local/go/src/n'}, 'http': {'method': 'POST', 'code': 400, 'timestamp': '2023-08-21 16:43:45.550851848 +0000 UTC m=+1664960.416353130', 'path': '/dbs/int/phys03/DBSWriter/bulkblocks', 'user_agent': 'DBSClient/Unknown/', 'x_forwarded_host': 'cmsweb-testbed.cern.ch:8443', 'x_forwarded_for': '137.138.159.63', 'remote_addr': '10.100.50.128:64004'}, 'exception': 400, 'type': 'HTTPError', 'message': 'DBSError Code:101 Description:DBS DB error Function:dbs.bulkblocks.InsertBulkBlocksConcurrently Message:unable to find parent lfn /store/data/Run2018D/ParkingBPH2/MINIAOD/05May2019promptD-v1/260005/E4DF44F0-8206-C84C-9BE2-80FD55210F3E.root Error: nested DBSError Code:103 Description:DBS DB query error, e.g. mailformed SQL statement Function:dbs.GetID Message: Error: sql: no rows in result set'}]

Relevant framgents from the above:

'x_forwarded_host': 'cmsweb-testbed.cern.ch:8443' 'path': '/dbs/int/phys03/DBSWriter/bulkblocks', Message: unable to find parent lfn /store/data/Run2018D/ParkingBPH2/MINIAOD/05May2019promptD-v1/260005/E4DF44F0-8206-C84C-9BE2-80FD55210F3E.root

But that file is there ! https://cmsweb-testbed.cern.ch/das/request?view=list&limit=50&instance=int%2Fglobal&input=file+file%3D%2Fstore%2Fdata%2FRun2018D%2FParkingBPH2%2FMINIAOD%2F05May2019promptD-v1%2F260005%2FE4DF44F0-8206-C84C-9BE2-80FD55210F3E.root+

error appears to be persistent.

I am playing with cmsweb-testbed/dbs/int/phys03 becasue I am testing some new code and it is difficult in global to find input datasets which have notbeen migrated already

[belforte]

here's one of the blocks which failed to be POSTed to /dbs/int/phys03/DBSWriter/bulkblocks failedblock.txt

[belforte]

same block was inserted finely in cmsweb-prod.cern.ch/dbs/prod/phys03

So I think nothing is wrong on my side nor on basics of dbs2go code, likely some misconfiguration or lack of generaility in the code, as I wrote the problem with sticking with prod DBS instance is that it is difficult to test migrations since most datasets have been migrated already.

[vkuznet]

Stefano, thanks for reporting the issue. Since production DBS works it means that in testbed db we do not have specific files , i.e. parents or children of some datasets. To fix the problem we may request full dump of production DBS again. It may also related to some other data ops activity, e.g. some files were invalidated. Until you really want this specific dataset for some reason I would leave everything as is and not spend time on it's debugging. Feel free to let us know what do you prefer, either close the ticket or continue.

Sent from Proton Mail mobile

-------- Original Message -------- On Aug 22, 2023, 4:38 AM, Stefano Belforte wrote:

same block was inserted finely in cmsweb-prod.cern.ch/dbs/prod/phys03

So I think nothing is wrong on my side nor on basics of dbs2go code, likely some misconfiguration or lack of generaility in the code, as I wrote the problem with sticking with prod DBS instance is that it is difficult to test migrations since most datasets have been migrated already.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[belforte]

thanks Valentin. Well, that file is also present in int/phys03, so AFAIU the block was properly migrated and then I really fail to understand how block insert could complain about lfn not found.

I will try other datasets !

[belforte]

Sorry for having attempted to blame DBS. I had a bug in my code and some needed blocks had not been migrated. Furthermore I confused looking in int/global vs. int/phys03. All is good. Thanks Valentin