search

JamesWoolfenden / sato

Sato converts ARM or CFN into Terraform
Apache License 2.0
104 stars 6 forks source link

add AWS::KinesisFirehose::DeliveryStream #16

Closed JamesWoolfenden closed 4 months ago

JamesWoolfenden commented 4 months ago

add AWS::KinesisFirehose::DeliveryStream

oijkn commented 4 months ago

I'm still encountering issues with this problem. Unfortunately, it's not working as expected. When I run the command, there are no messages in the logs and no files are being generated.

Could you please check this or provide additional guidance to help resolve the issue?

Thanks in advance.

JamesWoolfenden commented 4 months ago

what version are you using? Is it the same cf?

On Thu, 25 Jul 2024 at 11:04, oijkn @.***> wrote:

I'm still encountering issues with this problem. Unfortunately, it's not working as expected. When I run the command, there are no messages in the logs and no files are being generated.

Could you please check this or provide additional guidance to help resolve the issue?

Thanks in advance.

— Reply to this email directly, view it on GitHub https://github.com/JamesWoolfenden/sato/issues/16#issuecomment-2249956923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALDV4DYUCAKUJK5XTW2CS3ZODETRAVCNFSM6AAAAABK6WCNRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBZHE2TMOJSGM . You are receiving this because you modified the open/close state.Message ID: @.***>

oijkn commented 4 months ago

I'm using sato version v0.1.19 and here is the cf file :

Resources:
  SecHubEventsFirehoseDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: aws-sechub-logs-kinesis-delivery-stream
      DeliveryStreamType: DirectPut
      DeliveryStreamEncryptionConfigurationInput:
        KeyType: AWS_OWNED_CMK
      ExtendedS3DestinationConfiguration:
        BucketARN: !GetAtt S3Bucketsechubyashdatafirehose.Arn
        Prefix: !Sub "sechub_events/managementid=${AWS::AccountId}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"
        ErrorOutputPrefix: "aws-sechub-logs-firehose-error/"
        RoleARN: !GetAtt kinesisFirehoseRole.Arn
        CloudWatchLoggingOptions:
          Enabled: true
          LogGroupName: !Ref FirehoseLogGroup
          LogStreamName: !Ref FirehoseLogStream
        BufferingHints:
          IntervalInSeconds: 900
          SizeInMBs: 30
        CompressionFormat: "GZIP"
        ProcessingConfiguration:
          Enabled: true
          Processors:
            - Type: Lambda
              Parameters:
                - ParameterName: LambdaArn
                  ParameterValue: !GetAtt LambdaFunctiontransformationsechub.Arn
                - ParameterName: BufferIntervalInSeconds
                  ParameterValue: 600
                - ParameterName: BufferSizeInMBs
                  ParameterValue: 3
JamesWoolfenden commented 4 months ago

id switched off the error by mistake, but your cf is invalid its confusing quotes and ints: 

Resources:
  SecHubEventsFirehoseDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: aws-sechub-logs-kinesis-delivery-stream
      DeliveryStreamType: DirectPut
      DeliveryStreamEncryptionConfigurationInput:
        KeyType: AWS_OWNED_CMK
      ExtendedS3DestinationConfiguration:
        BucketARN: !GetAtt S3Bucketsechubyashdatafirehose.Arn
        Prefix: !Sub "sechub_events/managementid=${AWS::AccountId}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"
        ErrorOutputPrefix: "aws-sechub-logs-firehose-error/"
        RoleARN: !GetAtt kinesisFirehoseRole.Arn
        CloudWatchLoggingOptions:
          Enabled: true
          LogGroupName: !Ref FirehoseLogGroup
          LogStreamName: !Ref FirehoseLogStream
        BufferingHints:
          IntervalInSeconds: 900
          SizeInMBs: 30
        CompressionFormat: "GZIP"
        ProcessingConfiguration:
          Enabled: true
          Processors:
            - Type: Lambda
              Parameters:
                - ParameterName: LambdaArn
                  ParameterValue: !GetAtt LambdaFunctiontransformationsechub.Arn
                - ParameterName: BufferIntervalInSeconds
                  ParameterValue: "600"
                - ParameterName: BufferSizeInMBs
                  ParameterValue: "3"

ill release a new version with more verbose errors

oijkn commented 4 months ago

With last version 0.1.20 here is the log :

4:46PM INF goformation parse failure json: cannot unmarshal number into Go struct field Template.Resources of type string

Edit: if I use your cf file, sato 0.1.20 can convert the file

4:53PM INF Created /sato/aws_athena_workgroup.athenworkgroup.tf
4:53PM INF Created /sato/aws_athena_namedquery.securityhubathenaview.tf
4:53PM WRN AWS::Logs::LogStream not found
4:53PM INF Created /sato/aws_iam_role.iamroleflatjson1role.tf
4:53PM INF Created /sato/aws_iam_role.kinesisfirehoserole.tf
4:53PM INF Created /sato/aws_events_rule.eventrulesechubkinesisintegration.tf
4:53PM WRN AWS::Glue::Crawler not found
4:53PM INF Created /sato/aws_iam_managedpolicy.iammanagedpolicypolicyserviceroleawslambdabasicexecutionrole.tf
4:53PM INF Created /sato/aws_kinesisfirehose_deliverystream.sechubeventsfirehosedeliverystream.tf
4:53PM INF Created /sato/aws_iam_role.iamroleamazoneventbridgeinvokefirehose.tf
4:53PM INF Created /sato/aws_logs_loggroup.firehoseloggroup.tf
4:53PM INF Created /sato/aws_logs_loggroup.lambdaloggroup.tf
4:53PM INF Created /sato/aws_iam_role.securityhubgluecrawlerrole.tf
4:53PM INF Created /sato/aws_iam_managedpolicy.iammanagedpolicyserviceroleamazoneventbridgeinvokefirehose.tf
4:53PM INF Created /sato/aws_s3_bucket.s3bucketsechubyashdatafirehose.tf
4:53PM INF Created /sato/aws_lambda_function.lambdafunctiontransformationsechub.tf
4:53PM WRN AWS::Glue::Database not found

It seems that there are still warnings in the logs...

JamesWoolfenden commented 4 months ago

like i already said you need to modify your faultly cf like i showed in the 2 samples

On Thu, 25 Jul 2024 at 15:47, oijkn @.***> wrote:

With last version 0.1.20 here is the log :

4:46PM INF goformation parse failure json: cannot unmarshal number into Go struct field Template.Resources of type string

— Reply to this email directly, view it on GitHub https://github.com/JamesWoolfenden/sato/issues/16#issuecomment-2250528420, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALDV4HHFGP6Y7B5CC2I5FDZOEFYXAVCNFSM6AAAAABK6WCNRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJQGUZDQNBSGA . You are receiving this because you modified the open/close state.Message ID: @.***>