Closed github-actions[bot] closed 2 months ago
rustyjux
commented
2 months ago @ikethecoder, I was able to reproduce the error (against the test branch running locally in docker compose, run 01/01, then 19/02).
I tried switching parseJsonString in OrganizationController to use parseBlobString (which makes sense, since the blob can contain yaml or json) in https://github.com/bcgov/api-services-portal/tree/feature/org-activity-error.
rustyjux
commented
2 months ago Most recent changes did not reproduce failure when running 01/01 then 19/02, but resulted in a 500 error (instead of 422) on the org/activity when running the full suite - https://github.com/bcgov/api-services-portal/actions/runs/10710602531
rustyjux
commented
2 months ago Debug logging shows:
YAML parsing failed for key blob. Error: end of the stream or a document separator is expected (1:1)
1 | }
-----^
Failing content for key blob:
}
Record details:
{
"result": "success",
"message": "{actor} deleted {environment} environment",
"params": {
"actor": "Janis Smith",
"environment": "test"
},
"activityAt": "2024-09-09T23:50:21.144Z",
"blob": "{\"access\":[]}"
}Errors are occurring when trying to parse {"access":[]} as YAML. This is odd because this record has json as the type in the Blob table.
This blob content is generated to accompany delete environment activity. Similar other json blob content which is related to activities are for deleted namespaces ({"access":[],"serviceAccounts":[]}) but this wont ever fetched since the deleted namespaces will be omitted from the assorted namespaces.
Pulling the json blob entries from the test env on OCP, they are limited to delete env/namespace (though it looks like these will have non-null values if there were consumers or SAs for the env/namespace at time of deletion), along with ops_metrics_<something> records which can be very lengthy (1MB+)
Next steps:
rustyjux
commented
2 months ago Returned to using parseJsonString but tweaked the function to:
export const parseJsonString = (obj: any, keys: string[]) => {
console.warn('Entering parseJsonString with keys:', keys);
console.warn('Object to parse:', JSON.stringify(obj, null, 2));
Object.entries(obj).forEach(
([key, val]) => {
if (keys.includes(key)) {
console.warn(`Attempting to parse key: ${key}`);
console.warn(`Value before parsing: ${val}`);
try {
if (typeof val === 'string') {
obj[key] = JSON.parse(val);
console.warn(`Successfully parsed ${key}`);
} else {
console.warn(`Skipping parse for ${key}: value is not a string`);
}
} catch (error) {
console.error(`Error parsing ${key}:`, error);
console.error(`Problematic value:`, val);
}
} else if (val && typeof val === 'object') {
parseJsonString(val, keys);
}
}
);
console.warn('Exiting parseJsonString. Parsed object:', JSON.stringify(obj, null, 2));
return obj;
};This showed more details about why the 422 errors were occurring before beyond SyntaxError: Unexpected token 's'. The problem values are YAML, e.g.:
services:
- host: httpbin.org
name: cc-service-for-platform
...The updated function has an internal error, but still returns a 200 with an object like:
{
"id": "153",
"result": "completed",
"message": "{actor} {action} {entity} {message}",
"params": {
"action": "published",
"actor": "sa-gw-c5a9b-e0000000-6bddb4d48c1b",
"entity": "gateway configuration",
"result": "completed"
},
"activityAt": "2024-09-10T23:20:36.626Z",
"blob": "services:\n- host: httpbin.org\n name: cc-service-for-platform\n port: 443\n protocol: https\n retries: 0\n routes:\n - hosts:\n - cc-service-for-platform.api.gov.bc.ca\n https_redirect_status_code: 426\n methods:\n - GET\n name: cc-service-for-platform-route\n path_handling: v0\n paths:\n - /\n strip_path: false\n tags:\n - ns.gw-c5a9b\n tags:\n - ns.gw-c5a9b\n"
}Would this suffice? I need more info on why /activity in OrganizationController might diverge from GatewayController.
rustyjux
commented
2 months ago Clarity! From Aidan:
Basically Organization and Gateway should have the exact same logic to prepare the Activity info.. only difference is the filter criteria. And It's there for completeness and can come in handy for us as you can get activity for deleted gateways, whereas GatewayController will not show you activity after its been deleted.
The test on
GET /organizations/{org}/activitykeeps failing. This endpoint is new to api v3.Likely caused by a bug with the API since the response body along with the 422 is
"message": "Syntax Error Parsing JSON"In Cypress tests:
To reproduce:
docker compose, run tests in 01/01, then run 19/02Attached is a dump of the
Activitytable from postgres after these steps - dump_activity.csvLogs at the error show:
From logs, it looks like OrganzationController /activity hits the error on
.map((o) => parseJsonString(o, ['blob']));- https://github.com/bcgov/api-services-portal/blob/ff77d4a95521b7b971007e12fc0d917ed44e3c53/src/controllers/v3/OrganizationController.ts#L304-L305as part of
whereas NamespaceController /activity, which I was able to call passing all the gateways without any error, uses https://github.com/bcgov/api-services-portal/blob/ff77d4a95521b7b971007e12fc0d917ed44e3c53/src/controllers/v3/GatewayController.ts#L244