search

snowplow / enrich

Snowplow Enrichment jobs and library
https://snowplowanalytics.com
Other
21 stars 39 forks source link

enrich-nsq: add S3 and GCS dependencies #793

Closed benjben closed 1 year ago

benjben commented 1 year ago

enrich-nsq can't download artifacts that are stored on S3 or GCS.

Example error with S3:

java.lang.IllegalStateException: No client initialized to download s3://snowplow-hosted-assets-proprietary-eu-central-1/third-party/com.iab/ip_exclude_current_cidr.txt
--
at com.snowplowanalytics.snowplow.enrich.common.fs2.io.Clients.download(Clients.scala:40)
at com.snowplowanalytics.snowplow.enrich.common.fs2.Assets$.downloadAndHash(Assets.scala:259)
at com.snowplowanalytics.snowplow.enrich.common.fs2.Assets$.$anonfun$download$1(Assets.scala:205)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.CompileScope.$anonfun$open$2(CompileScope.scala:146)
at map @ fs2.internal.CompileScope.open(CompileScope.scala:137)
at flatMap @ fs2.internal.CompileScope.open(CompileScope.scala:141)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$15(Algebra.scala:533)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$14(Algebra.scala:531)
at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.FreeC$.$anonfun$compile$7(Algebra.scala:463)
at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:460)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
at flatten$extension @ com.snowplowanalytics.snowplow.enrich.common.fs2.Run$.$anonfun$run$2(Run.scala:139)
at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$1(CompileScope.scala:183)
at flatMap @ fs2.internal.CompileScope.acquireResource(CompileScope.scala:180)

We need that in snowplow-mini.