search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.26k stars 1.61k forks source link

Parse_json error #21515

Closed Askas00 closed 1 month ago

Askas00 commented 1 month ago

A note for the community

show me unable to pase json error

Problem

ERROR transform{component_kind="transform" component_id=remap component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (10:31): unable to parse json: EOF while parsing an object at line 1 column 4095" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

Configuration

log_schema:
  timestamp_key: "inserted_at"
  host_key: "vector_host"
  source_type_key: "vector_source"
sources:
  source:
    type: "stdin"

transforms:
  remap:
    type: remap
    drop_on_error: false
    inputs:
      - source
    source: |-
      .rawLog = parse_json!(.message)
      .inserted_at = now()
      structured,err = parse_json(.message)
      if err != null {
        log("Unable to parse json:" + err,level:"error")
        log(.,level:"error") 
      }
      .vector_host = "x.x.x.x"
      . = merge!(.rawLog,.)
sinks:
  out:
    inputs: [ "remap" ]
    type: "console"
    encoding:
      codec: "json"

Version

vector 0.37.1 (x86_64-unknown-linux-gnu cb6635a 2024-04-09 13:45:06.561412437)

Debug Output

No response

Example Data

{"@timestamp":"2024-10-13T14:15:05.000Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.8.1","_id":"38554172902259704816838881741112329325705085794567127040"},"env":"xxxxx","detail-type":"GuardDuty Finding","account":"123345555555","time":"2024-10-13T14:15:05Z","id":"ab29dd0b-ab1f-f396-e6f3-9ba5f0cb7122","awscloudwatch":{"log_group":"/aws/events/Guarduty","log_stream":"4fefd4c7-5578-3907-be9a-530e3b5d4b21","ingestion_time":"2024-10-13T14:15:05.000Z"},"region":"ap-northeast-1","detail":{"severity":2,"updatedAt":"2024-10-13T14:01:02.153Z","title":"An unprotected port on EC2 instance i-xxxxxxxxxxxxx is being probed.","schemaVersion":"2.0","region":"ap-northeast-1","arn":"arn:aws:guardduty:ap-northeast-1:123345555555:detector/6ebde52034b5991bbc8c5sdfsdfd14bfc19c59/finding/9cc6490b50c20825dc1617d60a896a44","accountId":"123345555555","partition":"aws","id":"9cc6490b50c20825dc1617d60a896a44","type":"Recon:EC2/PortProbeUnprotectedPort","service":{"action":{"portProbeAction":{"portProbeDetails":[{"remoteIpDetails":{"organization":{"asn":"398722","asnOrg":"CENSYS-ARIN-03","isp":"Censys-arin-03","org":"Censys-arin-03"},"country":{"countryName":"United States"},"city":{"cityName":""},"geoLocation":{"lat":37.751,"lon":-97.822},"ipAddressV4":"1.1.1.1"},"localPortDetails":{"port":8443,"portName":"HTTPS"}},{"localPortDetails":{"port":3001,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Amsterdam"},"geoLocation":{"lat":52.3759,"lon":4.8975},"ipAddressV4":"1.1.1.1","organization":{"asn":"60068","asnOrg":"Datacamp Limited","isp":"Datacamp","org":"Datacamp"},"country":{"countryName":"Netherlands"}}},{"localPortDetails":{"port":3000,"portName":"Unknown"},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"org":"China Telecom","asn":"58461","asnOrg":"CT-HangZhou-IDC","isp":"China Telecom"},"country":{"countryName":"China"},"city":{"cityName":""},"geoLocation":{"lat":34.7732,"lon":113.722}}},{"localPortDetails":{"port":3002,"portName":"Unknown"},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"},"country":{"countryName":"Austria"},"city":{"cityName":"Vienna"},"geoLocation":{"lat":48.1982,"lon":16.3917}}},{"localPortDetails":{"portName":"Unknown","port":18443},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"},"country":{"countryName":"Austria"},"city":{"cityName":"Vienna"},"geoLocation":{"lat":48.1982,"lon":16.3917}}}],"blocked":false},"actionType":"PORT_PROBE"},"resourceRole":"TARGET","archived":false,"count":18497,"serviceName":"guardduty","detectorId":"6ebde52034b5991bbc8c5sdfsdfd14bfc19c59","eventFirstSeen":"2023-12-22T15:07:46.000Z","eventLastSeen":"2024-10-13T13:56:33.000Z","additionalInfo":{"type":"default","threatListName":"ProofPoint","value":"{\"threatListName\":\"ProofPoint\"}"},"evidence":{"threatIntelligenceDetails":[{"threatNames":[],"threatListName":"ProofPoint"}]}},"resource":{"resourceType":"Instance","instanceDetails":{"iamInstanceProfile":null,"outpostArn":null,"tags":[{"key":"register_type","value":"auto_ec2"},{"key":"new-env","value":"xxxxx"},{"key":"Name","value":"xxxxx-Web-sisReport-3"},{"key":"map-migrated","value":"d-server-01i3oxwzuamvhn"},{"key":"new-app","value":"xxxxx"},{"value":"xxxxx","key":"new-dptm"}],"imageDescription":"Centos7 AMI v20231201","availabilityZone":"ap-northeast-1c","instanceId":"i-xxxxxxxxxxxxx","instanceType":"m6i.2xlarge","launchTime":"2023-12-12T07:54:49.000Z","platform":null,"productCodes":[{"productCodeId":"cvugziknvmxgqna9noibqnnsy","productCodeType":"marketplace"}],"networkInterfaces":[{"publicDnsName":"ecx-x-x-x-188.ap-northeast-1.compute.amazonaws.com","publicIp":"1.1.1.1","ipv6Addresses":[],"privateIpAddress":"1.1.1.1","privateIpAddresses":[{"privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1"}],"securityGroups":[{"groupName":"launch-wizard-2","groupId":"sg-xxxxxx"},{"groupName":"xxxxx","groupId":"sg-xxxxxx"},{"groupName":"xxxxx","groupId":"sg-xxxxxx"}],"networkInterfaceId":"eni-xxxxx","privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","subnetId":"subnet-xxxxxxxx","vpcId":"vpc-xxxxx"}],"instanceState":"running","imageId":"ami-xxxx"}},"createdAt":"2023-12-22T15:13:25.892Z","description":"An EC2 instance has an unprotected port which is being probed by a known malicious host."},"resources":[],"source":"aws.guardduty","log.file.path":"/aws/events/Guarduty/4fefd4c7-5578-3907-be9a-530e3b5d4b21","event":{"ingested":"2024-10-13T14:18:21.696Z","id":"38554172902259704816838881741112329325705085794567127040"},"version":"0"}

Additional Context

No response

References

No response

jorgehermo9 commented 1 month ago

Hi,

It seems that you are trying to parse the json twice


 .rawLog = parse_json!(.message)
      structured,err = parse_json(.message)

The first time, you are aborting your vrl program if an error is encountered.

The second time, you are handling the error inside of an if statement and logging the whole input.

I suggest to remove the first parse_json .rawLog = parse_json!(.message) so you can log the failing events

Askas00 commented 1 month ago

`log_schema: timestamp_key: "inserted_at" host_key: "vector_host" source_type_key: "vector_source" sources: source: type: "stdin"

transforms: remap: type: remap drop_on_error: false inputs:

same error

Askas00 commented 1 month ago

Is the string too long, and is there a limit on the maximum string length in JSON

jszwedko commented 1 month ago

The example input here seems to be invalid JSON:

jq: parse error: Invalid literal at line 1, column 2917

I'm not aware of any limits on max length for JSON parsing.

Askas00 commented 1 month ago

{"@timestamp":"2024-10-13T14:15:05.000Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.8.1","_id":"38554172902259704816838881741112329325705085794567127040"},"env":"xxxxx","detail-type":"GuardDuty Finding","account":"123345555555","time":"2024-10-13T14:15:05Z","id":"ab29dd0b-ab1f-f396-e6f3-9ba5f0cb7122","awscloudwatch":{"log_group":"/aws/events/Guarduty","log_stream":"4fefd4c7-5578-3907-be9a-530e3b5d4b21","ingestion_time":"2024-10-13T14:15:05.000Z"},"region":"ap-northeast-1","detail":{"severity":2,"updatedAt":"2024-10-13T14:01:02.153Z","title":"An unprotected port on EC2 instance i-xxxxxxxxxxxxx is being probed.","schemaVersion":"2.0","region":"ap-northeast-1","arn":"arn:aws:guardduty:ap-northeast-1:123345555555:detector/6ebde52034b5991bbc8c5sdfsdfd14bfc19c59/finding/9cc6490b50c20825dc1617d60a896a44","accountId":"123345555555","partition":"aws","id":"9cc6490b50c20825dc1617d60a896a44","type":"Recon:EC2/PortProbeUnprotectedPort","service":{"action":{"portProbeAction":{"portProbeDetails":[{"remoteIpDetails":{"organization":{"asn":"398722","asnOrg":"CENSYS-ARIN-03","isp":"Censys-arin-03","org":"Censys-arin-03"},"country":{"countryName":"United States"},"city":{"cityName":""},"geoLocation":{"lat":37.751,"lon":-97.822},"ipAddressV4":"1.1.1.1"},"localPortDetails":{"port":8443,"portName":"HTTPS"}},{"localPortDetails":{"port":3001,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Amsterdam"},"geoLocation":{"lat":52.3759,"lon":4.8975},"ipAddressV4":"1.1.1.1","organization":{"asn":"60068","asnOrg":"Datacamp Limited","isp":"Datacamp","org":"Datacamp"},"country":{"countryName":"Netherlands"}}},{"localPortDetails":{"port":3000,"portName":"Unknown"},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"org":"China Telecom","asn":"58461","asnOrg":"CT-HangZhou-IDC","isp":"China Telecom"},"country":{"countryName":"China"},"city":{"cityName":""},"geoLocation":{"lat":34.7732,"lon":113.722}}},{"localPortDetails":{"port":3002,"portName":"Unknown"},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"},"country":{"countryName":"Austria"},"city":{"cityName":"Vienna"},"geoLocation":{"lat":48.1982,"lon":16.3917}}},{"localPortDetails":{"portName":"Unknown","port":18443},"remoteIpDetails":{"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"},"country":{"countryName":"Austria"},"city":{"cityName":"Vienna"},"geoLocation":{"lat":48.1982,"lon":16.3917}}}],"blocked":false},"actionType":"PORT_PROBE"},"resourceRole":"TARGET","archived":false,"count":18497,"serviceName":"guardduty","detectorId":"6ebde52034b5991bbc8c5sdfsdfd14bfc19c59","eventFirstSeen":"2023-12-22T15:07:46.000Z","eventLastSeen":"2024-10-13T13:56:33.000Z","additionalInfo":{"type":"default","threatListName":"ProofPoint","value":"{\"threatListName\":\"ProofPoint\"}"},"evidence":{"threatIntelligenceDetails":[{"threatNames":[],"threatListName":"ProofPoint"}]}},"resource":{"resourceType":"Instance","instanceDetails":{"iamInstanceProfile":null,"outpostArn":null,"tags":[{"key":"register_type","value":"auto_ec2"},{"key":"new-env","value":"xxxxx"},{"key":"Name","value":"xxxxx-Web-sisReport-3"},{"key":"map-migrated","value":"d-server-01i3oxwzuamvhn"},{"key":"new-app","value":"xxxxx"},{"value":"xxxxx","key":"new-dptm"}],"imageDescription":"Centos7 AMI v20231201","availabilityZone":"ap-northeast-1c","instanceId":"i-xxxxxxxxxxxxx","instanceType":"m6i.2xlarge","launchTime":"2023-12-12T07:54:49.000Z","platform":null,"productCodes":[{"productCodeId":"cvugziknvmxgqna9noibqnnsy","productCodeType":"marketplace"}],"networkInterfaces":[{"publicDnsName":"ecx-x-x-x-188.ap-northeast-1.compute.amazonaws.com","publicIp":"1.1.1.1","ipv6Addresses":[],"privateIpAddress":"1.1.1.1","privateIpAddresses":[{"privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1"}],"securityGroups":[{"groupName":"launch-wizard-2","groupId":"sg-xxxxxx"},{"groupName":"xxxxx","groupId":"sg-xxxxxx"},{"groupName":"xxxxx","groupId":"sg-xxxxxx"}],"networkInterfaceId":"eni-xxxxx","privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","subnetId":"subnet-xxxxxxxx","vpcId":"vpc-xxxxx"}],"instanceState":"running","imageId":"ami-xxxx"}},"createdAt":"2023-12-22T15:13:25.892Z","description":"An EC2 instance has an unprotected port which is being probed by a known malicious host."},"resources":[],"source":"aws.guardduty","log.file.path":"/aws/events/Guarduty/4fefd4c7-5578-3907-be9a-530e3b5d4b21","event":{"ingested":"2024-10-13T14:18:21.696Z","id":"38554172902259704816838881741112329325705085794567127040"},"version":"0"}

Askas00 commented 1 month ago

Example data encountered an error while replacing sensitive data

Askas00 commented 1 month ago
image
jszwedko commented 1 month ago

Hmm, I'm not able to reproduce this with the given config and given example input 🙁 . I get, as output,:

{"@metadata":{"_id":"38554172902259704816838881741112329325705085794567127040","beat":"filebeat","type":"_doc","version":"8.8.1"},"@timestamp":"2024-10-13T14:15:05.000Z","account":"123345555555","awscloudwatch":{"ingestion_time":"2024-10-13T14:15:05.000Z","log_group":"/aws/events/Guarduty","log_stream":"4fefd4c7-5578-3907-be9a-530e3b5d4b21"},"detail":{"accountId":"123345555555","arn":"arn:aws:guardduty:ap-northeast-1:123345555555:detector/6ebde52034b5991bbc8c5sdfsdfd14bfc19c59/finding/9cc6490b50c20825dc1617d60a896a44","createdAt":"2023-12-22T15:13:25.892Z","description":"An EC2 instance has an unprotected port which is being probed by a known malicious host.","id":"9cc6490b50c20825dc1617d60a896a44","partition":"aws","region":"ap-northeast-1","resource":{"instanceDetails":{"availabilityZone":"ap-northeast-1c","iamInstanceProfile":null,"imageDescription":"Centos7 AMI  v20231201","imageId":"ami-xxxx","instanceId":"i-xxxxxxxxxxxxx","instanceState":"running","instanceType":"m6i.2xlarge","launchTime":"2023-12-12T07:54:49.000Z","networkInterfaces":[{"ipv6Addresses":[],"networkInterfaceId":"eni-xxxxx","privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1","privateIpAddresses":[{"privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1"}],"publicDnsName":"ecx-x-x-x-188.ap-northeast-1.compute.amazonaws.com","publicIp":"1.1.1.1","securityGroups":[{"groupId":"sg-xxxxxx","groupName":"launch-wizard-2"},{"groupId":"sg-xxxxxx","groupName":"xxxxx"},{"groupId":"sg-xxxxxx","groupName":"xxxxx"}],"subnetId":"subnet-xxxxxxxx","vpcId":"vpc-xxxxx"}],"outpostArn":null,"platform":null,"productCodes":[{"productCodeId":"cvugziknvmxgqna9noibqnnsy","productCodeType":"marketplace"}],"tags":[{"key":"register_type","value":"auto_ec2"},{"key":"new-env","value":"xxxxx"},{"key":"Name","value":"xxxxx-Web-sisReport-3"},{"key":"map-migrated","value":"d-server-01i3oxwzuamvhn"},{"key":"new-app","value":"xxxxx"},{"key":"new-dptm","value":"xxxxx"}]},"resourceType":"Instance"},"schemaVersion":"2.0","service":{"action":{"actionType":"PORT_PROBE","portProbeAction":{"blocked":false,"portProbeDetails":[{"localPortDetails":{"port":8443,"portName":"HTTPS"},"remoteIpDetails":{"city":{"cityName":""},"country":{"countryName":"United States"},"geoLocation":{"lat":37.751,"lon":-97.822},"ipAddressV4":"1.1.1.1","organization":{"asn":"398722","asnOrg":"CENSYS-ARIN-03","isp":"Censys-arin-03","org":"Censys-arin-03"}}},{"localPortDetails":{"port":3001,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Amsterdam"},"country":{"countryName":"Netherlands"},"geoLocation":{"lat":52.3759,"lon":4.8975},"ipAddressV4":"1.1.1.1","organization":{"asn":"60068","asnOrg":"Datacamp Limited","isp":"Datacamp","org":"Datacamp"}}},{"localPortDetails":{"port":3000,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":""},"country":{"countryName":"China"},"geoLocation":{"lat":34.7732,"lon":113.722},"ipAddressV4":"1.1.1.1","organization":{"asn":"58461","asnOrg":"CT-HangZhou-IDC","isp":"China Telecom","org":"China Telecom"}}},{"localPortDetails":{"port":3002,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Vienna"},"country":{"countryName":"Austria"},"geoLocation":{"lat":48.1982,"lon":16.3917},"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"}}},{"localPortDetails":{"port":18443,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Vienna"},"country":{"countryName":"Austria"},"geoLocation":{"lat":48.1982,"lon":16.3917},"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"}}}]}},"additionalInfo":{"threatListName":"ProofPoint","type":"default","value":"{\"threatListName\":\"ProofPoint\"}"},"archived":false,"count":18497,"detectorId":"6ebde52034b5991bbc8c5sdfsdfd14bfc19c59","eventFirstSeen":"2023-12-22T15:07:46.000Z","eventLastSeen":"2024-10-13T13:56:33.000Z","evidence":{"threatIntelligenceDetails":[{"threatListName":"ProofPoint","threatNames":[]}]},"resourceRole":"TARGET","serviceName":"guardduty"},"severity":2,"title":"An unprotected port on EC2 instance i-xxxxxxxxxxxxx is being probed.","type":"Recon:EC2/PortProbeUnprotectedPort","updatedAt":"2024-10-13T14:01:02.153Z"},"detail-type":"GuardDuty Finding","env":"xxxxx","event":{"id":"38554172902259704816838881741112329325705085794567127040","ingested":"2024-10-13T14:18:21.696Z"},"id":"ab29dd0b-ab1f-f396-e6f3-9ba5f0cb7122","inserted_at":"2024-10-16T16:48:22.362496Z","log.file.path":"/aws/events/Guarduty/4fefd4c7-5578-3907-be9a-530e3b5d4b21","message":"{\"@timestamp\":\"2024-10-13T14:15:05.000Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.8.1\",\"_id\":\"38554172902259704816838881741112329325705085794567127040\"},\"env\":\"xxxxx\",\"detail-type\":\"GuardDuty Finding\",\"account\":\"123345555555\",\"time\":\"2024-10-13T14:15:05Z\",\"id\":\"ab29dd0b-ab1f-f396-e6f3-9ba5f0cb7122\",\"awscloudwatch\":{\"log_group\":\"/aws/events/Guarduty\",\"log_stream\":\"4fefd4c7-5578-3907-be9a-530e3b5d4b21\",\"ingestion_time\":\"2024-10-13T14:15:05.000Z\"},\"region\":\"ap-northeast-1\",\"detail\":{\"severity\":2,\"updatedAt\":\"2024-10-13T14:01:02.153Z\",\"title\":\"An unprotected port on EC2 instance i-xxxxxxxxxxxxx is being probed.\",\"schemaVersion\":\"2.0\",\"region\":\"ap-northeast-1\",\"arn\":\"arn:aws:guardduty:ap-northeast-1:123345555555:detector/6ebde52034b5991bbc8c5sdfsdfd14bfc19c59/finding/9cc6490b50c20825dc1617d60a896a44\",\"accountId\":\"123345555555\",\"partition\":\"aws\",\"id\":\"9cc6490b50c20825dc1617d60a896a44\",\"type\":\"Recon:EC2/PortProbeUnprotectedPort\",\"service\":{\"action\":{\"portProbeAction\":{\"portProbeDetails\":[{\"remoteIpDetails\":{\"organization\":{\"asn\":\"398722\",\"asnOrg\":\"CENSYS-ARIN-03\",\"isp\":\"Censys-arin-03\",\"org\":\"Censys-arin-03\"},\"country\":{\"countryName\":\"United States\"},\"city\":{\"cityName\":\"\"},\"geoLocation\":{\"lat\":37.751,\"lon\":-97.822},\"ipAddressV4\":\"1.1.1.1\"},\"localPortDetails\":{\"port\":8443,\"portName\":\"HTTPS\"}},{\"localPortDetails\":{\"port\":3001,\"portName\":\"Unknown\"},\"remoteIpDetails\":{\"city\":{\"cityName\":\"Amsterdam\"},\"geoLocation\":{\"lat\":52.3759,\"lon\":4.8975},\"ipAddressV4\":\"1.1.1.1\",\"organization\":{\"asn\":\"60068\",\"asnOrg\":\"Datacamp Limited\",\"isp\":\"Datacamp\",\"org\":\"Datacamp\"},\"country\":{\"countryName\":\"Netherlands\"}}},{\"localPortDetails\":{\"port\":3000,\"portName\":\"Unknown\"},\"remoteIpDetails\":{\"ipAddressV4\":\"1.1.1.1\",\"organization\":{\"org\":\"China Telecom\",\"asn\":\"58461\",\"asnOrg\":\"CT-HangZhou-IDC\",\"isp\":\"China Telecom\"},\"country\":{\"countryName\":\"China\"},\"city\":{\"cityName\":\"\"},\"geoLocation\":{\"lat\":34.7732,\"lon\":113.722}}},{\"localPortDetails\":{\"port\":3002,\"portName\":\"Unknown\"},\"remoteIpDetails\":{\"ipAddressV4\":\"1.1.1.1\",\"organization\":{\"asn\":\"202616\",\"asnOrg\":\"Wien Energie GmbH\",\"isp\":\"Wien Energie\",\"org\":\"Wien Energie\"},\"country\":{\"countryName\":\"Austria\"},\"city\":{\"cityName\":\"Vienna\"},\"geoLocation\":{\"lat\":48.1982,\"lon\":16.3917}}},{\"localPortDetails\":{\"portName\":\"Unknown\",\"port\":18443},\"remoteIpDetails\":{\"ipAddressV4\":\"1.1.1.1\",\"organization\":{\"asn\":\"202616\",\"asnOrg\":\"Wien Energie GmbH\",\"isp\":\"Wien Energie\",\"org\":\"Wien Energie\"},\"country\":{\"countryName\":\"Austria\"},\"city\":{\"cityName\":\"Vienna\"},\"geoLocation\":{\"lat\":48.1982,\"lon\":16.3917}}}],\"blocked\":false},\"actionType\":\"PORT_PROBE\"},\"resourceRole\":\"TARGET\",\"archived\":false,\"count\":18497,\"serviceName\":\"guardduty\",\"detectorId\":\"6ebde52034b5991bbc8c5sdfsdfd14bfc19c59\",\"eventFirstSeen\":\"2023-12-22T15:07:46.000Z\",\"eventLastSeen\":\"2024-10-13T13:56:33.000Z\",\"additionalInfo\":{\"type\":\"default\",\"threatListName\":\"ProofPoint\",\"value\":\"{\\\"threatListName\\\":\\\"ProofPoint\\\"}\"},\"evidence\":{\"threatIntelligenceDetails\":[{\"threatNames\":[],\"threatListName\":\"ProofPoint\"}]}},\"resource\":{\"resourceType\":\"Instance\",\"instanceDetails\":{\"iamInstanceProfile\":null,\"outpostArn\":null,\"tags\":[{\"key\":\"register_type\",\"value\":\"auto_ec2\"},{\"key\":\"new-env\",\"value\":\"xxxxx\"},{\"key\":\"Name\",\"value\":\"xxxxx-Web-sisReport-3\"},{\"key\":\"map-migrated\",\"value\":\"d-server-01i3oxwzuamvhn\"},{\"key\":\"new-app\",\"value\":\"xxxxx\"},{\"value\":\"xxxxx\",\"key\":\"new-dptm\"}],\"imageDescription\":\"Centos7 AMI  v20231201\",\"availabilityZone\":\"ap-northeast-1c\",\"instanceId\":\"i-xxxxxxxxxxxxx\",\"instanceType\":\"m6i.2xlarge\",\"launchTime\":\"2023-12-12T07:54:49.000Z\",\"platform\":null,\"productCodes\":[{\"productCodeId\":\"cvugziknvmxgqna9noibqnnsy\",\"productCodeType\":\"marketplace\"}],\"networkInterfaces\":[{\"publicDnsName\":\"ecx-x-x-x-188.ap-northeast-1.compute.amazonaws.com\",\"publicIp\":\"1.1.1.1\",\"ipv6Addresses\":[],\"privateIpAddress\":\"1.1.1.1\",\"privateIpAddresses\":[{\"privateDnsName\":\"ip-x-x-x-x.ap-northeast-1.compute.internal\",\"privateIpAddress\":\"1.1.1.1\"}],\"securityGroups\":[{\"groupName\":\"launch-wizard-2\",\"groupId\":\"sg-xxxxxx\"},{\"groupName\":\"xxxxx\",\"groupId\":\"sg-xxxxxx\"},{\"groupName\":\"xxxxx\",\"groupId\":\"sg-xxxxxx\"}],\"networkInterfaceId\":\"eni-xxxxx\",\"privateDnsName\":\"ip-x-x-x-x.ap-northeast-1.compute.internal\",\"subnetId\":\"subnet-xxxxxxxx\",\"vpcId\":\"vpc-xxxxx\"}],\"instanceState\":\"running\",\"imageId\":\"ami-xxxx\"}},\"createdAt\":\"2023-12-22T15:13:25.892Z\",\"description\":\"An EC2 instance has an unprotected port which is being probed by a known malicious host.\"},\"resources\":[],\"source\":\"aws.guardduty\",\"log.file.path\":\"/aws/events/Guarduty/4fefd4c7-5578-3907-be9a-530e3b5d4b21\",\"event\":{\"ingested\":\"2024-10-13T14:18:21.696Z\",\"id\":\"38554172902259704816838881741112329325705085794567127040\"},\"version\":\"0\"}","rawLog":{"@metadata":{"_id":"38554172902259704816838881741112329325705085794567127040","beat":"filebeat","type":"_doc","version":"8.8.1"},"@timestamp":"2024-10-13T14:15:05.000Z","account":"123345555555","awscloudwatch":{"ingestion_time":"2024-10-13T14:15:05.000Z","log_group":"/aws/events/Guarduty","log_stream":"4fefd4c7-5578-3907-be9a-530e3b5d4b21"},"detail":{"accountId":"123345555555","arn":"arn:aws:guardduty:ap-northeast-1:123345555555:detector/6ebde52034b5991bbc8c5sdfsdfd14bfc19c59/finding/9cc6490b50c20825dc1617d60a896a44","createdAt":"2023-12-22T15:13:25.892Z","description":"An EC2 instance has an unprotected port which is being probed by a known malicious host.","id":"9cc6490b50c20825dc1617d60a896a44","partition":"aws","region":"ap-northeast-1","resource":{"instanceDetails":{"availabilityZone":"ap-northeast-1c","iamInstanceProfile":null,"imageDescription":"Centos7 AMI  v20231201","imageId":"ami-xxxx","instanceId":"i-xxxxxxxxxxxxx","instanceState":"running","instanceType":"m6i.2xlarge","launchTime":"2023-12-12T07:54:49.000Z","networkInterfaces":[{"ipv6Addresses":[],"networkInterfaceId":"eni-xxxxx","privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1","privateIpAddresses":[{"privateDnsName":"ip-x-x-x-x.ap-northeast-1.compute.internal","privateIpAddress":"1.1.1.1"}],"publicDnsName":"ecx-x-x-x-188.ap-northeast-1.compute.amazonaws.com","publicIp":"1.1.1.1","securityGroups":[{"groupId":"sg-xxxxxx","groupName":"launch-wizard-2"},{"groupId":"sg-xxxxxx","groupName":"xxxxx"},{"groupId":"sg-xxxxxx","groupName":"xxxxx"}],"subnetId":"subnet-xxxxxxxx","vpcId":"vpc-xxxxx"}],"outpostArn":null,"platform":null,"productCodes":[{"productCodeId":"cvugziknvmxgqna9noibqnnsy","productCodeType":"marketplace"}],"tags":[{"key":"register_type","value":"auto_ec2"},{"key":"new-env","value":"xxxxx"},{"key":"Name","value":"xxxxx-Web-sisReport-3"},{"key":"map-migrated","value":"d-server-01i3oxwzuamvhn"},{"key":"new-app","value":"xxxxx"},{"key":"new-dptm","value":"xxxxx"}]},"resourceType":"Instance"},"schemaVersion":"2.0","service":{"action":{"actionType":"PORT_PROBE","portProbeAction":{"blocked":false,"portProbeDetails":[{"localPortDetails":{"port":8443,"portName":"HTTPS"},"remoteIpDetails":{"city":{"cityName":""},"country":{"countryName":"United States"},"geoLocation":{"lat":37.751,"lon":-97.822},"ipAddressV4":"1.1.1.1","organization":{"asn":"398722","asnOrg":"CENSYS-ARIN-03","isp":"Censys-arin-03","org":"Censys-arin-03"}}},{"localPortDetails":{"port":3001,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Amsterdam"},"country":{"countryName":"Netherlands"},"geoLocation":{"lat":52.3759,"lon":4.8975},"ipAddressV4":"1.1.1.1","organization":{"asn":"60068","asnOrg":"Datacamp Limited","isp":"Datacamp","org":"Datacamp"}}},{"localPortDetails":{"port":3000,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":""},"country":{"countryName":"China"},"geoLocation":{"lat":34.7732,"lon":113.722},"ipAddressV4":"1.1.1.1","organization":{"asn":"58461","asnOrg":"CT-HangZhou-IDC","isp":"China Telecom","org":"China Telecom"}}},{"localPortDetails":{"port":3002,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Vienna"},"country":{"countryName":"Austria"},"geoLocation":{"lat":48.1982,"lon":16.3917},"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"}}},{"localPortDetails":{"port":18443,"portName":"Unknown"},"remoteIpDetails":{"city":{"cityName":"Vienna"},"country":{"countryName":"Austria"},"geoLocation":{"lat":48.1982,"lon":16.3917},"ipAddressV4":"1.1.1.1","organization":{"asn":"202616","asnOrg":"Wien Energie GmbH","isp":"Wien Energie","org":"Wien Energie"}}}]}},"additionalInfo":{"threatListName":"ProofPoint","type":"default","value":"{\"threatListName\":\"ProofPoint\"}"},"archived":false,"count":18497,"detectorId":"6ebde52034b5991bbc8c5sdfsdfd14bfc19c59","eventFirstSeen":"2023-12-22T15:07:46.000Z","eventLastSeen":"2024-10-13T13:56:33.000Z","evidence":{"threatIntelligenceDetails":[{"threatListName":"ProofPoint","threatNames":[]}]},"resourceRole":"TARGET","serviceName":"guardduty"},"severity":2,"title":"An unprotected port on EC2 instance i-xxxxxxxxxxxxx is being probed.","type":"Recon:EC2/PortProbeUnprotectedPort","updatedAt":"2024-10-13T14:01:02.153Z"},"detail-type":"GuardDuty Finding","env":"xxxxx","event":{"id":"38554172902259704816838881741112329325705085794567127040","ingested":"2024-10-13T14:18:21.696Z"},"id":"ab29dd0b-ab1f-f396-e6f3-9ba5f0cb7122","log.file.path":"/aws/events/Guarduty/4fefd4c7-5578-3907-be9a-530e3b5d4b21","region":"ap-northeast-1","resources":[],"source":"aws.guardduty","time":"2024-10-13T14:15:05Z","version":"0"},"region":"ap-northeast-1","resources":[],"source":"aws.guardduty","time":"2024-10-13T14:15:05Z","vector_host":"x.x.x.x","vector_source":"stdin","version":"0"}
Askas00 commented 1 month ago

What version is vector

Askas00 commented 1 month ago

I'll try the latest version

jszwedko commented 1 month ago

I tested with v0.41.1.

Askas00 commented 1 month ago

0.41.1 also reports an error, what is your YAML configuration

jszwedko commented 1 month ago

0.41.1 also reports an error, what is your YAML configuration

I used the config you provided above:

log_schema:
  timestamp_key: "inserted_at"
  host_key: "vector_host"
  source_type_key: "vector_source"
sources:
  source:
    type: "stdin"

transforms:
  remap:
    type: remap
    drop_on_error: false
    inputs:
      - source
    source: |-
      .rawLog = parse_json!(.message)
      .inserted_at = now()
      structured,err = parse_json(.message)
      if err != null {
        log("Unable to parse json:" + err,level:"error")
        log(.,level:"error") 
      }
      .vector_host = "x.x.x.x"
      . = merge!(.rawLog,.)
sinks:
  out:
    inputs: [ "remap" ]
    type: "console"
    encoding:
      codec: "json"
jszwedko commented 1 month ago

Closing since this doesn't seem to be a bug. Feel free to open a GitHub Discussion if you are still having trouble, though.