Open dominik-kornas-wttech opened 1 year ago
JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-1851
Thank you for the report, we'll investigate. The leading slash is the correct way to reference the alarm package AFAIK.
Our daily e2e tests for aio-lib-runtime passed when creating a feed, but only tests for /whisk.system/alarms/alarm
. Will continue probing and add a test for /whisk.system/alarms/once
Our e2e tests were run 12 hours ago, I just ran them manually and there are 2 failing tests with triggers and feeds - so it might be a service issue, not a cli issue. Will update this thread.
Thank you for quick response and looking into this! Looking forward to seeing results of your investigation.
If I may add, there is an odd behavior that I observed, which may be related to this issue.
When I accidentally tried to create a trigger with a date in the past, I have received an error (as expected) informing me about that. But it also resulted in one more error, with no details in the returned json (see below). Just a plain app error.
Since then, all triggers stopped firing as they were scheduled. I can still fire them manually from the command line, but they are not triggered on scheduled time or interval. Like some internal scheduler/cron has been broken by my attempt to set a trigger in the past. What's also important, that issue fixes itself overnight (I've experienced that twice), presumably by some runtime maintenance that (I guess) restarts something in the service.
Details
This is how it appears on the activation list as:
Datetime Status Kind Version Activation ID Start Wait Init Duration Entity
─────────────── ───────── ───────── ──────── ─ ──────────────────────────────── ───── ──── ──── ──────── ───────────────────────────────
10/26 15:50:19 app error nodejs:10 0.0.245 fc47163f11a1433087163f11a1f330e5 warm 25 0 1001ms whisk.system/alarms/once
10/26 15:50:19 app error nodejs:10 0.0.245 d3114b406618461b914b406618b61bf7 warm 50 0 70ms whisk.system/alarms/once
$ aio rt activation get d3114b406618461b914b406618b61bf7
{
"actionHost": "10.216.0.31",
"activationId": "d3114b406618461b914b406618b61bf7",
"annotations": [
{
"key": "path",
"value": "whisk.system/alarms/once"
},
{
"key": "waitTime",
"value": 50
},
{
"key": "kind",
"value": "nodejs:10"
},
{
"key": "timeout",
"value": false
},
{
"key": "limits",
"value": {
"concurrency": 1,
"logs": 10,
"memory": 256,
"timeout": 60000
}
}
],
"duration": 70,
"end": 1666792219086,
"invokerInstanceId": {
"instance": 4,
"instanceType": "invoker",
"uniqueName": "rt-invoker-4",
"userMemory": "1500000000000000 B"
},
"logs": [],
"name": "once",
"namespace": "***************",
"podName": "wskrt-invoker-44-376762-prewarm-nodejs10",
"publish": false,
"response": {
"result": {
"error": {
"error": "date parameter '2022-10-26T13:16:00.000Z' must be in the future"
}
},
"size": 85,
"status": "application error",
"success": false
},
"start": 1666792219016,
"subject": "***************",
"version": "0.0.245"
}
$ aio rt activation get fc47163f11a1433087163f11a1f330e5
{
"actionHost": "10.216.0.31",
"activationId": "fc47163f11a1433087163f11a1f330e5",
"annotations": [
{
"key": "path",
"value": "whisk.system/alarms/once"
},
{
"key": "waitTime",
"value": 25
},
{
"key": "kind",
"value": "nodejs:10"
},
{
"key": "timeout",
"value": false
},
{
"key": "limits",
"value": {
"concurrency": 1,
"logs": 10,
"memory": 256,
"timeout": 60000
}
}
],
"duration": 1001,
"end": 1666792220317,
"invokerInstanceId": {
"instance": 4,
"instanceType": "invoker",
"uniqueName": "rt-invoker-4",
"userMemory": "1500000000000000 B"
},
"logs": [],
"name": "once",
"namespace": "***************",
"podName": "wskrt-invoker-44-376762-prewarm-nodejs10",
"publish": false,
"response": {
"result": {
"error": {}
},
"size": 12,
"status": "application error",
"success": false
},
"start": 1666792219316,
"subject": "***************",
"version": "0.0.245"
}
The service issue turned out to be a red-herring. I tried creating an alarm via the aio
cli and it was successful just now.
The next step is to get the value of the x-request-id
header of the failing call.
DEBUG=*
before your command, then run it$env:DEBUG = '*'
, then run your commandIf I may add, there is an odd behavior that I observed, which may be related to this issue.
When I accidentally tried to create a trigger with a date in the past, I have received an error (as expected) informing me about that. But it also resulted in one more error, with no details in the returned json (see below). Just a plain app error.
There are sometimes issues with the alarm provider's state. Right now the best course of action is to delete all rules and triggers and re-create them. Not ideal, but if you want to use a manifest that would be painless.
Hi @shazron, your mention of Windows Powershell actually struck me with the obvious question about the shell that's being used. I am mostly using git-bash and the failing command was executed from git-bash. When I tried in Powershell and and then also in CMD it worked properly!
So apparently there is something incorrect with git-bash that I wasn't aware of.
If that's of any help, here's the failing git-bash command with --verbose
that includes x-request-id
.
I've removed some parts of the output to make it more readable.
$ wsk trigger create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:00:00.000Z" -p deleteAfterFire "rules" --verbose
REQUEST:
[PUT] https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger?overwrite=false
Req Headers
{
(...)
"User-Agent": [
"OpenWhisk-CLI/1.0 (2021-04-01T23:49:54.523+0000) windows 386"
]
}
Req Body
{"name":"my-trigger","annotations":[{"key":"feed","value":"/_/once"}]}
RESPONSE:Got response with code 200
Resp Headers
{
"X-Azure-Ref": [
"0QchbYwAAAACXtTcq3m8oS7aWYqgq/sCMV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
],
"X-Cache": [
"CONFIG_NOCACHE"
],
"X-Request-Id": [
"OECo2knrvXfcI3z62I9vJnXAUONRvhpH"
]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"53890-381goldflyingfish-stage","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
REQUEST:
[POST] https://adobeioruntime.net/api/v1/namespaces/_/actions/./C:/Program%20Files/Git/whisk.system/alarms/once?blocking=true&result=false
Req Headers
(...)
Req Body
{"authKey":"********","date":"2023-10-28T13:00:00.000Z","deleteAfterFire":"rules","lifecycleEvent":"CREATE","triggerName":"/_/my-trigger"}
RESPONSE:Got response with code 400
Resp Headers
{
(...)
"X-Azure-Ref": [
"0Q8hbYwAAAAASxLjiWO9ARZ2RO6eChBhaV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
],
"X-Cache": [
"CONFIG_NOCACHE"
],
"X-Request-Id": [
"aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ"
]
}
Response body size is 105 bytes
Response body received:
{"code":"aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ","error":"The name of the entity contains illegal characters."}
REQUEST:
[GET] https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger
Req Headers
(...)
RESPONSE:Got response with code 200
Resp Headers
{
(...)
"X-Azure-Ref": [
"0Q8hbYwAAAAAjVmuQ+KGzS5Urp7XPGnliV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
],
"X-Cache": [
"CONFIG_NOCACHE"
],
"X-Request-Id": [
"RTkPcuZQ7WvPgGN8hojo4rpWRCXpkGTJ"
]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"*********","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
REQUEST:
[POST] https://adobeioruntime.net/api/v1/namespaces/_/actions/once?blocking=true&result=false
Req Headers
(...)
Req Body
{"authKey":"********","date":"2023-10-28T13:00:00.000Z","deleteAfterFire":"rules","lifecycleEvent":"DELETE","triggerName":"/_/my-trigger"}
RESPONSE:Got response with code 404
Resp Headers
{
(...)
"X-Azure-Ref": [
"0Q8hbYwAAAADlRMiIxz7CR5TeqMqGDYnkV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
],
"X-Cache": [
"CONFIG_NOCACHE"
],
"X-Request-Id": [
"1Q4vHlh2cvtnXFO0IlZPG5QgbsqDM3da"
]
}
Response body size is 92 bytes
Response body received:
{"code":"1Q4vHlh2cvtnXFO0IlZPG5QgbsqDM3da","error":"The requested resource does not exist."}
REQUEST:
[DELETE] https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger
Req Headers
(...)
RESPONSE:Got response with code 200
Resp Headers
{
(...)
"X-Azure-Ref": [
"0Q8hbYwAAAADVvSKWEwegTacCFNRNvkqSV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
],
"X-Cache": [
"CONFIG_NOCACHE"
],
"X-Request-Id": [
"FO2cKNeQriGaVGaWjipZ0rWca1D80I2F"
]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"********","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
ok: deleted trigger my-trigger
error: Unable to create trigger 'my-trigger': Unable to configure feed '/_/once': The name of the entity contains illegal characters. (code aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ)
There are sometimes issues with the alarm provider's state. Right now the best course of action is to delete all rules and triggers and re-create them. Not ideal, but if you want to use a manifest that would be painless.
I tried exactly this previously without success and had to wait until the next day when it fixed automatically. I will give it another try, though.
I found that setting the variable MSYS_NO_PATHCONV=1 (actually to any value) through export MSYS_NO_PATHCONV=1
or simply prepending it before the command, solves the issue and makes both following commands execute properly under git-bash.
MSYS_NO_PATHCONV=1 wsk trigger create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:30:00.000Z" -p deleteAfterFire "rules"
MSYS_NO_PATHCONV=1 aio rt create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:30:00.000Z" -p deleteAfterFire "rules"
So it appears to be an issue caused by my insufficient knowledge of git-bash.1
Running MSYS_NO_PATHCONV=1 aio app run
in git-bash deploys all triggers with feeds included in the manifest properly.
However the breaking scheduler issue stays. I've created a trigger with date in the past through the manifest and since then all triggers stopped firing. Even when I removed all trigger and rules, and re-created them again.
Oh wow! I should have asked for the output of aio info
as well to figure out your shell.
Does double quoting "/whisk.system/alarms/once" work, without the MSYS_NO_PATHCONV=1
env var?
I'll write up this path issue as a separate issue for other users' benefit (and to prime Google search).
However the breaking scheduler issue stays. I've created a trigger with date in the past through the manifest and since then all triggers stopped firing. Even when I removed all trigger and rules, and re-created them again.
Don't have an answer for you yet for this issue.
@dominik-kornas-wttech here's the advice I was given:
We still have the known issue when trying to delete and recreate the triggers of the same name. This requires a lot of rework in the current alarms and is a low priority for now -- the mitigation is to simply use unique names for the triggers to prevent reuse.
Hi! I tried with double quotes, but the results is the same: an error prompt informing about feed name not being a valid one.
And regarding trigger names, I'm sure I tried with different ones and the issue persisted. From my point of view, this isn't really an issue with recreating triggers. The initial attempt to create a trigger fails because of the date in the past. Then, even if I create another trigger with a valid date and a different name, this and any other triggers aren't fired.
Though, I don't know the inner details of the alarms package so it still may be somehow related to the mentioned known issue, just the observation doesn't seem to be that much related.
Describe the bug Creating a trigger that has a feed (any of the alarms package feeds) fails with a
Feed name is not valid
error.Example command failing with an error
namespace and APi key obfuscated
This looks like related to a leading slash, as the same command executed with
wsk cli
works:Trying to use
aio cli trigger create
without the leading slash generates another errorThe requested resource does not exist
Note that in addition the working command does not align with the documentation which instructs to provide leading slash. Documentation examples:
Expected behavior The
aio cli trigger create
works as documented. The App Builder app deployment (aio app run
) works as well and allows to create triggers along actions during deployment.Versions: