search

HEPCloud / decisionengine

HEPCloud Decision Engine framework
Apache License 2.0
6 stars 25 forks source link

Need way to poll sources (from redis) #627

Open StevenCTimm opened 2 years ago

StevenCTimm commented 2 years ago

When everything is now concentrated into one channel and that channel goes offline for some reason, then it is not possible to poll any of the data blocks via the de-client --print-product functionality. In this case it would be nice to be able to poll the data from the running sources in the redis facility, because the sources all keep running.

StevenCTimm commented 2 years ago

Discussion in today's meeting brought up that there is likely a way already in redis-cli to do this, we just have to figure out the right commands. redis-cli can be run via podman exec against the container.

knoepfel commented 2 years ago

@StevenCTimm, we can likely find a way to print the value of enqueued data products. However, now that message-bloat problem with DE 2.0 has been resolved, printing the value of enqueued data products will only be useful when no channel is able to consume the products--e.g., the channel has gone offline for some reason. So...is this facility still desired?

StevenCTimm commented 2 years ago

The case where the channel has gone offline is exactly the place where we need to examine the contents of the source products. One of the key reasons that channels do go offline is because of either the absence of info that should be there, or extra junk coming in that should not be there. Source polling in this case is important.. and the read_redis.py temporary utility that was made I've found to be quite useful.

Steve

From: Kyle Knoepfel @.> Sent: Monday, March 28, 2022 11:31 AM To: HEPCloud/decisionengine @.> Cc: Steven C Timm @.>; Mention @.> Subject: Re: [HEPCloud/decisionengine] Need way to poll sources (from redis) (Issue #627)

@StevenCTimmhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_StevenCTimm&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=mlZYJErCQdJdQXLfiwvJxXpohyQcOkZ3Rl3h2WHt2xk&e=, we can likely find a way to print the value of enqueued data products. However, now that message-bloat problem with DE 2.0 has been resolved, printing the value of enqueued data products will only be useful when no channel is able to consume the products--e.g., the channel has gone offline for some reason. So...is this facility still desired?

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HEPCloud_decisionengine_issues_627-23issuecomment-2D1080876428&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=0Z7gninVZIANW87A32MmfAofV1kg3zeQtxbXLGufKVU&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGG4SOC7OYZ7TEUNITC7Q53VCHNHNANCNFSM5QHTQDJA&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=10BCTK25QMgkMYibLRbpYg&m=etnG_Pc54mEO5vpJNtYFZPVhjp0sHLbhUcIHLjc7oVTeLmQZamQezYal_oevFugW&s=B656u_cHO-YZftxYPujabsKtWJK7MagK8PpUPjs-rn4&e=. You are receiving this because you were mentioned.Message ID: @.***>