search

mozilla-services / foxsec-pipeline

Log analysis pipeline utilizing Apache Beam
Mozilla Public License 2.0
25 stars 9 forks source link

Persist Guardduty alerts to either Bigquery or GCS #207

Closed ajvb closed 5 years ago

ajvb commented 5 years ago

As we discussed, it would be helpful to persist all Guardduty alerts in there raw json to either a separate BigQuery table or to GCS.

The goal is to be able to dig into alerts easily during an investigation.

adrianosela commented 5 years ago

The "payload" field in my alerts is still empty (null), would this be a good place to throw the raw events? @ajvb

cc:// @ameihm0912

ajvb commented 5 years ago

@adrianosela No, I think that it should be persisted outside of the context of foxsec-pipeline alerts. To be more explicit, we should persist the com.amazonaws.services.guardduty.model.Finding parsed within

https://github.com/mozilla-services/foxsec-pipeline/blob/49e28bf903515dc48ebe7767709ee947c9710927/src/main/java/com/mozilla/secops/parser/GuardDuty.java#L11-L22

to it's own BigQuery table or to GCS.

adrianosela commented 5 years ago

Gotcha, will implement a "Persistor" transform for each of ETD and GD. Will take the target BQ as an argument i.e. --persistETDFindingsBigQuery and --persistGDFindingsBigQuery and store them separately.

Does this sound like a reasonable approach? @ajvb @ameihm0912

Also, what should the behavior be if the user fails to provide these arguments? Run anyways and not persist, or die on deploy?

ajvb commented 5 years ago

Let's just do Guardduty. For ETD, we can use CSCC.

I'd base it off of https://github.com/mozilla-services/foxsec-pipeline/blob/master/src/main/java/com/mozilla/secops/OutputOptions.java#L23-L26 and https://github.com/mozilla-services/foxsec-pipeline/blob/master/src/main/java/com/mozilla/secops/CompositeOutput.java#L88-L107

So you should probably just be able to use org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO for this.

adrianosela commented 5 years ago

Closing in favor of https://github.com/mozilla-services/foxsec-pipeline/issues/229 @ajvb