search

awslabs / kinesis-agent-windows

An extensible Windows agent that ingests logs and metrics to AWS services such as Kinesis Stream, Kinesis Firehose, CloudWatch Logs and CloudWatch.
Apache License 2.0
39 stars 22 forks source link

Unable to connect source and sink #37

Closed PranathiSG closed 1 year ago

PranathiSG commented 1 year ago

I want to stream data present in a .txt file from local folder to kinesis stream using kinesis agent.

I have installed the kinesis agent and configured appsettings.json file.

When I start the service , I get following error

2023-02-08 16:24:01.8310 default ERROR Unable to load event source myKinesisSource System.NullReferenceException: Object reference not set to an instance of an object. at Amazon.KinesisTap.Filesystem.FileSystemFactory.CreateFsSource(IPlugInContext context) at Amazon.KinesisTap.Filesystem.FileSystemFactory.Amazon.KinesisTap.Core.IFactory.CreateInstance(String entry, IPlugInContext context) at Amazon.KinesisTap.Hosting.Session.LoadEventSources() 2023-02-08 16:24:01.8666 default ERROR Unable to connect source myKinesisSource to sink myKinesisStreamSink. 2023-02-08 16:24:01.8722 default ERROR SourceRef myKinesisSource not found for pipe id myKinesisStreamPipe

appsettings.json configuration file:

{ "Sources": [ { "Id": "myKinesisSource", "SourceType": "DirectorySource", "Directory": "C:\sales_file\", "FileNameFilter": "*.txt" } ], "Sinks": [ { "Id": "myKinesisStreamSink", "SinkType": "KinesisStream", "StreamName": streamname, "AccessKey": accesskey, "SecretKey": secretkey, "Region": region } ], "Pipes": [ { "Id": "myKinesisStreamPipe", "SourceRef": "myKinesisSource", "SinkRef": "myKinesisStreamSink" } ], "Telemetrics": { "off": "true" } }

I'm new to kinesis streaming and data agent concept. Any help on this is highly appreciated.

dhhoang commented 1 year ago

@PranathiSG , Please add a RecordParser config for your source declaration. For collecting entire line-by-line logs you can use the SingleLine parser e.g.

{
  "Id": "myKinesisSource",
  "SourceType": "DirectorySource",
  "Directory": "C:\\sales_file\\",
  "FileNameFilter": "*.txt",
  "RecordParser": "SingleLine"
}

See more info on the DirectorySource here: https://docs.aws.amazon.com/kinesis-agent-windows/latest/userguide/source-object-declarations.html#directory-source-configuration

dhhoang commented 1 year ago

@PranathiSG, I'm closing this. Feel free to comment if you have any issues.