`vagrant up` fails on macOS at `npm install` step

[angelsk]

Fails on npm step. Also a bunch of warnings about Python 2.7 being deprecated and various options, but that doesn't stop it working up to the npm step.

macOS Big Sur version 11.6.2 Vagrant 2.2.16 snowplow-mini master@8889e11a8c7ed44093ce6efacf561d68d128f3fc

npm-debug.log

[paulboocock]

Hi @angelsk

Thanks for the report. This seems to only affect Macs, but you can fix it with some manual intervention. We do intend to deprecate and/or update the UI in future plans for Snowplow Mini but it's a little way down our roadmap.

Snowplow Mini works best directly deployed to AWS or GCP rather than being used locally and is its main purpose. Snowplow Micro is a simple snowplow pipeline for testing tracking locally that we typically recommend.

Anyway, to fix this locally, you first need to install node (and therefore npm on your machine) from https://nodejs.org/ Then if you open a terminal window and navigate to provisioning/resources/ui, you can run npm install. This should suceed. If you then navigate back to the root of the Snowplow Mini repo and run vagrant up your set up should hopefully complete.

[angelsk]

Thanks @paulboocock. I need to test enrichments, which I'm fairly certain don't work with Micro, and I didn't want to add cost to our AWS just to test something that I could test locally (and is a one-off test). I appreciate that this can be spun up in vagrant, as it makes it really easy (well except when Mac does a silly).

[angelsk]

Ok, so that doesn't work out of the box either.... (I re-ran with --force)

ui % npm install
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! While resolving: snowplow-mini@0.1.0
npm ERR! Found: react@15.6.1
npm ERR! node_modules/react
npm ERR!   react@"15.6.1" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer react@"0.12.x - 1.x" from react-simpletabs@0.7.0
npm ERR! node_modules/react-simpletabs
npm ERR!   react-simpletabs@"^0.7.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

[paulboocock]

Ah, I think this is because you are using npm v7+ which is bundled with the most recent version of node, and which had a behaviour change. I'm using Node 14 LTS on my machine and it appears to work for me.

I think either trying npm install --legacy-peer-deps which should use npm v6 behaviours, or downgrading to Node 14 LTS on your machine, should help.

[angelsk]

Yup - we're using:

% npm -v
7.20.6
% node -v
v14.17.1

[angelsk]

However, I've now found the posts re: ip_lookups enrichment being disabled in Snowplow Mini.... which is what I wanted to test because I have no idea what permissions are needed on the DB and I didn't want to make it public, because of the MaxMind policy. We have the db in a private bucket rn....

[istreeter]

I need to test enrichments, which I'm fairly certain don't work with Micro

I have often wondered whether Micro should have a feature to run enrichments. Technically it would not be hard to add this to micro. I wonder what kind of use cases this would solve, or how many people would need this feature.

Sorry I know that's not directly relevant to the original question.

[angelsk]

I need to test enrichments, which I'm fairly certain don't work with Micro

I have often wondered whether Micro should have a feature to run enrichments. Technically it would not be hard to add this to micro. I wonder what kind of use cases this would solve, or how many people would need this feature.

Sorry I know that's not directly relevant to the original question.

Technically it is, coz that's why I needed to install Mini ;) It would be awesome to be able to test enrichments with Micro!

[oguzhanunlu]

Hey @angelsk ,

On the permission error, you can comment these lines to workaround the permission error. After it builds, you can revert your change and execute vagrant reload to sync those files.

On maxmind testing, as shared in our docs, URI of the database can start with either http(s) or s3 or gs as the scheme but not file. (assuming your local test would refer to the db downloaded to your disk)

Enabling enrich to be able to refer to local maxmind artifacts could be discussed. For the moment, the best option looks like a cloud deployment for large variant to keep the cost at minimum.

[oguzhanunlu]

(assuming your local test would refer to the db downloaded to your disk)

In case this assumption is wrong, you should be able to point your enrichment config at a private s3 or gcs bucket as long as you setup AWS/GCP credentials on the host Mini is running on. (in this case, you should setup your cloud credentials within virtual machine), edit enrich config to use those credentials, and add ip lookup enrichment config

[oguzhanunlu]

Looking at the stream-enrich-nsq codebase in more detail, it is able to download objects from private buckets from either S3 or GCS as long as enrich is configured.

• vagrant ssh to browse into the machine, edit enrich config at /home/ubuntu/snowplow/configs/snowplow-stream-enrich.hocon based on this section.
• Setup your AWS/GCP credentials within virtual machine based on the section I reference above.
• Put your ip lookup enrichment config in /home/ubuntu/snowplow/configs/enrichments
• Restart enrich container to take these changes into account

[angelsk]

(assuming your local test would refer to the db downloaded to your disk)

In case this assumption is wrong, you should be able to point your enrichment config at a private s3 or gcs bucket as long as you setup AWS/GCP credentials on the host Mini is running on. (in this case, you should setup your cloud credentials within virtual machine), edit enrich config to use those credentials, and add ip lookup enrichment config

I did say it's in a private bucket. I was trying to figure out how to give the enrich module access to it, because there's nothing in the docs about that. This is why I wanted to test.

I don't want to install Snowplow Mini in AWS/GCP. And I've used the Terraform modules to set up my production Pipeline - so config in hocon is set to iam. https://discourse.snowplowanalytics.com/t/question-about-ip-lookups-enrichment-and-storage-of-maxmind-db-in-our-s3/6300

[oguzhanunlu]

because there's nothing in the docs about that.

You're right, we didn't document how Mini can access private maxmind artifacts from localhost. I hope my latest message can help in that regard.

[angelsk]

because there's nothing in the docs about that.

You're right, we didn't document how Mini can access private maxmind artifacts from localhost. I hope my latest message can help in that regard.

Sorry I wasn't clear. There's nothing in the docs about how to allow it from production pipeline :) even though it says to put it in a private bucket

It also doesn't help that I'm using the Terraform modules for the open source pipeline. I'll figure something out.

[oguzhanunlu]

Sorry for the misunderstanding on my side, I thought you wanted to build Mini on localhost and test maxmind enrichment using a private db artifact.

I'll reply to your discourse post to continue there as it is not Mini related anymore.

[angelsk]

Sorry for the misunderstanding on my side, I thought you wanted to build Mini on localhost and test maxmind enrichment using a private db artifact.

I'll reply to your discourse post to continue there as it is not Mini related anymore.

I did originally, but then I realised the issue would still be permissions :). As I'm confident of the JSON config. Sorry for the confusion and I appreciate your help.

[oguzhanunlu]

No worries at all and I'm closing this issue as it seems no further activity is required. Thanks for sharing @angelsk !