Closed lucasjellema closed 1 year ago
misraved
commented
1 year ago Thanks @lucasjellema for raising this issue 👍.
The analysis makes sense. We need to check if the corresponding API for log entries supports pagination or not. If yes, then it is safe to create a new table for it.
Does terraform have any references to the log entries?
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
bigdatasourav
commented
1 year ago @lucasjellema, sorry for the delayed response.
We are actively investigating and conducting thorough testing of the above API using various test cases. You can find all the relevant details here. We welcome your feedback and suggestions as we continue working on this.
bigdatasourav
commented
1 year ago Based on the documentation and my investigation of the API, I have compiled the following findings: [branch link]
BuildCompartementRegionList matrix.{
"id": "7d3999af-0e11-4920-bd92-b92c66c7ca66",
"data": {
"message": "ListVolumes succeeded",
"request": {
"id": "C28A04E2C9834CF59C61F7E30ABD4947/827F74513B00798A39D5347515342F25/F09F6AB771178A4F8F000EDC98C0E0CA",
"path": "/20160918/volumes",
"action": "GET",
"headers": {
"Date": [
"Sun, 09 Jul 2023 23:17:35 GMT"
],
"Accept": [
"application/json"
],
"Connection": [
"keep-alive"
],
"User-Agent": [
"Oracle-JavaSDK/2.54.1 (Linux/4.14.35-2047.525.1.el7uek.x86_64; Java/1.8.0_301; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.3/25.301-b09-jvmci-20.3-b18)"
],
"Authorization": [
"Signature headers=\"date (request-target) host\",keyId=\"ST$eyJraWQiOiJhc3dfb2MxX2o0eGQiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJvY2ktb3B0aW1pemVyIiwic3ZjIjoib2NpLW9wdGltaXp
50Ijoib2NpZDEudGVuYW5jeS5vYzEuLmFhYWFhiVPU_ip-OaF7vE3m1wTWHnIBcsc8P-D0xGmRL_MihTcM_-2JD5aqrowlXgysZqt9cqdiAuMIK6QdfivgNk23X736Ccw99nFA84mu1_IuPSjg8jIbLw8jqhOqSmVrbjZ1ZJgl9CLO-AzwEQ8
],
"opc-request-id": [
"C28A04E2C9834CF59C61F7E30ABD4947"
],
"opc-client-info": [
"Oracle-JavaSDK/2.54.1"
],
"opc-client-retries": [
"true"
]
},
"parameters": {
"limit": [
"1000"
],
"compartmentId": [
"ocid1.compartment.oc1..aaaaaaaaiawn27wsv4npbf6g3eiobdbhhletvekk2us3ftgm6uq"
]
}
},
"identity": {
"authType": "service",
"callerId": null,
"tenantId": "ocid1.tenancy.oc1..aaaaaaaa64d4rod2cgrqleyskgao4wkbeosylqnlrbgka",
"ipAddress": "192.29.165.245",
"userAgent": "Oracle-JavaSDK/2.54.1 (Linux/4.14.35-2047.525.1.el7uek.x86_64; Java/1.8.0_301; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.3/25.301-b09-jvmci-20.3-b18)",
"callerName": null,
"credentials": "ST$eyJraWQiOiJhc3dfb2MxXdiAuMIK6QdfivgNk23X736Ccw99nFA84mu1_IuPSjg8jIbLw8jqhOqSmVrbjZ1ZJgl9CLO-AzwEQ8cYR7z0gARH53Wkwkiy7plglgfDl385oGjK8l-XnySP1Fxnoo1T
"principalId": "oci-optimizer/C7:7D:61:03:28:7D:76:43:A3:97:4B:23:9E:FD:D7:78:8E:01:53:13:5F:D2:46:EF:56:59:38:F3:8F:F9:F6:14",
"principalName": "oci-optimizer",
"consoleSessionId": null
},
"response": {
"status": "200",
"headers": {
"Date": [
"Sun, 09 Jul 2023 23:17:36 GMT"
],
"Content-Type": [
"application/json"
],
"Content-Length": [
"2"
],
"opc-request-id": [
"C28A04E2C9834CF59C61F7E30ABD4947/827F74513B00798A39D5347515342F25/F09F6AB771178A4F8F000EDC98C0E0CA"
],
"X-Content-Type-Options": [
"nosniff"
]
},
"message": null,
"payload": {
},
"responseTime": "2023-07-09T23:17:36.866Z"
},
"eventName": "ListVolumes",
"resourceId": null,
"definedTags": null,
"stateChange": {
"current": null,
"previous": null
},
"freeformTags": null,
"compartmentId": "ocid1.compartment.oc1..aaaaaaaaiawn27wsv4npbf6g3eiobdkgfb",
"compartmentName": "sub-compartment-raj-test-compartment",
"eventGroupingId": "C28A04E2C9834CF59C61F7E30ABD4947/827F74513B00798A39D5347515342F25",
"additionalDetails": {
"X-Real-Port": 51618
},
"availabilityDomain": "AD2"
},
"time": "2023-07-09T23:17:36.866Z",
"type": "com.oraclecloud.BlockVolumes.ListVolumes",
"oracle": {
"tenantid": "ocid1.tenancy.oc1..aaaaaaaa64d4rod2cgrqleyskgao4wkbeosy",
"loggroupid": "_Audit",
"ingestedtime": "2023-07-09T23:17:38.866Z",
"compartmentid": "ocid1.compartment.oc1..aaaaaaaaiawn27wsv4npbf6g3eio"
},
"source": "",
"dataschema": "2.0",
"specversion": "1.0"
}@cbruno10 please provide your thoughts.
cbruno10
commented
1 year ago @bigdatasourav Thanks for doing research into the API, my responses are below:
timestamp column, similar to the https://hub.steampipe.io/plugins/turbot/aws/tables/aws_cloudwatch_log_event table.start_time and the current time as end_time?bigdatasourav
commented
1 year ago LogContent response structure is not consistent -
{
"id": "c45ef0a9-2be2-4365-b957-1087af3f5ed0",
"data": {
"isPar": false,
"region": "us-ashburn-1",
"apiType": "native",
"endTime": "2023-07-12T04:10:23.230Z",
"message": "List of Objects retrieved.",
"bucketId": "ocid1.bucket.oc1.iad.aaaaaaaab7flmnz4bs7itdxureoguhucq5ihbm6whc35h4lpa",
"tenantId": "ocid1.tenancy.oc1..aaaaaaaahnm7gleh5soecxzjetci3yjjnjqmfh2q37cyljaq",
"startTime": "2023-07-12T04:10:23.204Z",
"userAgent": "Oracle-GoSDK/65.28.0 (linux/amd64; go/go1.19.10)",
"bucketName": "bucket-20230710-1243",
"statusCode": 200,
"tenantName": "turbot",
"credentials": "ocid1.tenancy.oc1..aaaaaaaahnm7gleh5soecxzjetci3yjjnjqmfk2q37cyljaq/ocid1.user.oc1..aaaaaaaaydv22bk4pk5uwk6boif7mbbvy7kvb6sb572cpsvpitq/4a:c
"principalId": "ocid1.user.oc1..aaaaaaaaydv22bk4p5hh7x3pcvk5uwk6boif7mbbvy72cpsvpitq",
"opcRequestId": "iad-1:al2nT_y2mbXJ8ZVeNju-NaK4kM_ta3OMWm3f6rEnMwr-KQtIOMTb018byk4Y27GP",
"bucketCreator": "ocid1.user.oc1..aaaaaaaazvio4hcyouxn2zd5latufffwyuwja7mwuj3x6pq4i2q",
"compartmentId": "ocid1.tenancy.oc1..aaaaaaaahnm7gleh5soecxzjetci3yjjnjqmfk4p4h2q37cyljaq",
"namespaceName": "bmqeqvslavsz",
"principalName": "SteampipeResourceListBot",
"requestAction": "GET",
"clientIpAddress": "104.45.202.149",
"compartmentName": "turbot",
"authenticationType": "user",
"requestResourcePath": "/n/bmqeqvslavsz/b/bucket-20230710-1243/o?fields=name%2Csize%2Cetag%2CtimeCreated%2Cmd5%2CtimeModified%2CstorageTier%2CarchivalState&limit=1000"
},
"time": "2023-07-12T04:10:23.230Z",
"type": "com.oraclecloud.objectstorage.listobjects",
"oracle": {
"logid": "ocid1.log.oc1.iad.amaaaaaa6igdexaahbhu27du77j6ptteohx57m7pj6fpn5m6cq",
"tenantid": "ocid1.tenancy.oc1..aaaaaaaahnm7gleh5soecxzjetci3yjjnjqmp4h2q37cyljaq",
"loggroupid": "ocid1.loggroup.oc1.iad.amaaaaaa6igdexaa2xbvm6hxx22nelt7j7w3c5sekrlwp7z5spvzrq",
"ingestedtime": "2023-07-12T04:10:44.503Z",
"compartmentid": "ocid1.tenancy.oc1..aaaaaaaahnm7gleh5soecxzjetci3yjjnjqmfkr4p4h2q37cyljaq"
},
"source": "bucket-20230710-1243",
"specversion": "1.0"
}We are going to design the table below way -
timestamp to support start and end time.
Is your feature request related to a problem? Please describe. I was looking at oci_logging_log and I noticed it does not cover the actual contents of log files. I was wondering if perhaps there can be support for queries against log-file contents.
I was checking out the Oracle supported Grafana plugin https://github.com/oracle/oci-grafana-logs that provides such support through Grafana and all of a sudden it seems doable. Specifically file https://github.com/oracle/oci-grafana-logs/blob/main/pkg/datasource.go provides quite a bit of insight into how this could be done
Describe the solution you'd like The ability to select lines from logs that satisfy conditions (regarding log , start and end timestamp, level and search string)
Whether this should be a new table or be integrated into oci_logging_log I am note. However, while writing this down, perhaps a new table does make most sense - something like oci_logging_log_entries.
Some references:
https://github.com/oracle/oci-go-sdk/blob/v65.28.0/loggingsearch/loggingsearch_logsearch_client.go, https://pkg.go.dev/github.com/oracle/oci-go-sdk/v65@v65.28.0/loggingsearch#SearchLogsResponse https://github.com/oracle/oci-grafana-logs/blob/main/pkg/datasource.go