Scale8 - Tag Manager & Data Manager

:eyes: *MIT License is applied to Platforms to enable network distribution. The rest of the project remains under AGPL v3.

A simple, yet powerful Google Tag Manager alternative that is fully open-sourced and privacy-friendly. Our in-built Google Analytics alternative doesn't use any cookies and is fully GDPR, CCPA and PECR compliant. We've made it quick and easy to try a demo or simply host it yourself in production. We'll also be offering a cloud hosted version in the near future.

The Data Manager provides pluggable backends to send data anywhere, and we've got a pipeline to increase both connectors and features. In just a few clicks, design a schema, configure a back-end and create an endpoint to send data to. The Tag Manager uses this same underlying tech to provide analytics.

Our mission has been to create an intuitive Tag Management tool that can be fully extended via custom platforms (built by our community!). We're a small team that is passionate about building great open-source software and hopefully reducing the market share of much larger companies. We are not a fan of increasing corporate or government surveillance, and we fundamentally believe that your website's users deserve better - this however needs to be balanced with the business requirement to monetise a property and personal data being shared.

Running the project

Here is a one-line example of how to get started if you already have Docker Compose installed.

~> curl -L https://github.com/scale8/scale8/raw/master/docker-compose.yml | docker-compose -f - up

Once the command above has started the services, point your browser at http://127.0.0.1:8080. You will be prompted to enter some basic details to configure the project.

Or if you have checked out the project already and installed the dependencies via ~> yarn install:all and wish to run it locally: -

~> yarn run:all

Platform structure

We've architected the entire platform to support docker and limited where possible complex build processes and mounting of extra volumes. We're scripting hard to provide complete one-click production ready setups for both AWS, Google Cloud and custom kubernetes setups.

Component	Description
API	Contains all the business logic and provides GraphQL / RESTFul endpoints that power the UI
Common	A small common library that is shared between project components
Edge	Designed to scale horizontally, the edge server is responsible for delivering platforms at the edge and also tracking data in
Platforms	Custom platforms created by the community and Scale8 that extend the functionality of the Tag Manager
Router	A simple router, created using Nginx and designed for spinning up the self-hosted version quickly
UI	Provides a static build of the UI using Next.js & React

Development - Install dependencies and build

Install all project dependencies: ~> yarn install:all
Build all projects: ~> yarn build:all

Analytics-only mode

As not everyone has the requirement to use a full Tag Manager, we've also provided an option to generate an extremely light version that only requires a few lines of JavaScript.

We have also provided full support for tracking Single Page Applications (SPAs) and hash-routing. These can be enabled when you install your application.

Features overview

Tag Manager

Tags: Create tags that automatically load, or include them on the fly in your application.
Rule Groups: Tags can be associated with one or more Rule Groups that can be run in parallel.
Rules: Rule groups can contain multiple rules with the first rule winning. Rules can also be repeated once they've won.
Events: Attach events to rules as triggers. Once attached they will listen for your chosen event to occur before allowing a rule to proceed.
Conditions & Exceptions: Attach conditions and exceptions to rules. These will test your data containers / data layers.
Actions: Once events and/or conditions are satisfied, then trigger one or more actions. App-state enables actions to trigger other actions.
Action Groups: Action groups contain actions and provide a useful container to separate logic.
Action Group Distributions: Run experiments (A/B tests) or monitor just a percentage of traffic. Easily distribute your actions across your users.
Installed Platforms: Install 3rd party platforms submitted by our community. These platforms extend Events, Data Containers and Actions.
Installed Platform Revisions: A revision is locked to a platform revision. Upgrade and deploy with confidence as connect platforms change.
Environments: Create as many environments as your team requires, matching your existing workflows.
Revision Control: All changes happen within a new revision. Preview and test without complex release cycles. Deploy with confidence.
Debugging: Debug your application with ease. Remote debugging coming soon!
Analytics: A simple and intuitive set of analytics tools. Event tracking with Event Group support.
Analytics-only Mode: Run an incredibly lightweight snippet of JavaScript code to gain full analytics data.

Data Manager

Ingest Endpoints: Generate new endpoints to send data to in just a few clicks.
Data Structure: Compose a data structure that is compatible with any backend we offer. Design once, use everywhere.
Revisions: Everything sits under revision control. Release with confidence.
Usage Monitoring: Track endpoint requests and data usage.
Environments: Test your latest data structure without compromising your production setup. Roll-back in seconds.

All products

Team Support: Invite team members and manage access levels.
Full API Support: Automate everything - The UI is static and calls the GraphQL API for everything.

Core concepts

We've done our best to simplify the process of organising and managing tags across one or more web properties. The Tag Manager is based on an event-driven, rule-based engine whereby a rule is triggered when Events and Conditions and Exceptions are all met, resulting in one or more Actions being taken.

Events

A triggered event within the browser can be easily managed by our simple interface. A clicked link, a page coming into focus, or a submitted form are all examples of events you may wish to listen to.

Conditions & Exceptions

A condition is a test made on a data container and aims to assert that some key belonging to a data container object is aligned with the expected value. An exception aims to make the same test, but instead exclude the rule should the test pass.

Actions

Upon meeting the rule's required events and condition, an action will be fired. Learn more about actions here.

Tech stack

TLDR;

UI: Next.js, TypeScript, React, Apollo, GraphQL, Material UI.
Edge: Micronaut, Java, Maven.
Router: Nginx - Used for self-hosted version only and not required for all setups.
API: InversifyJS, Node, TypeScript, Express, MongoDB, Apollo, GraphQL, Jest.
Platforms: TypeScript, Webpack, Jest.

A longer explanation

We originally started off with a TypeScript front-end and Scala driven back-end. To provide the code transparency we wanted, coupled with a decision to open-source the project required us to move to almost purely TypeScript with just the edge server in Java. We fundamentally believe the future of this project sits with the open-source community now and there are more TypeScript and Java engineers than Scala engineers.

Just before releasing we've moved from CRA to Next.js too. We've also introduced inversion of control (IOC) in our TypeScript back-end. Our goal is for this project to be as extensible as possible. A plug and play model provides us with the opportunity to provide multiple services and prevent lock-ins.

container.bind<BaseStorage>(TYPES.BackendStorage).to(MongoDBStorage).inSingletonScope();
container.bind<BaseDatabase>(TYPES.BackendDatabase).to(MongoDb).inSingletonScope();
container.bind<BaseLogger>(TYPES.BackendLogger).to(ConsoleLogger).inSingletonScope();
container.bind<BaseEmail>(TYPES.BackendEmail).to(Mailer).inSingletonScope();
container.bind<BaseConfig>(TYPES.BackendConfig).to(EnvironmentConfig).inSingletonScope();

We've taken a similar approach with our Java code too.

@Replaces(StorageInterface.class)
@Singleton
@Requires(property = "backend-storage", value = "google")
public class GoogleStorage implements StorageInterface {
    ...
}

At present, we provide only MongoDB and Google's BigQuery for analytics. MongoDB is only suitable for small projects and for the purpose of demo / testing. We have full Clickhouse support in our yet to be released cloud version, and we'll be adding this support in the self-hosted version shortly along with Redshift and Postgres.

We've also removed our hourly aggregation pipelines from the self-hosted version. This added an unnecessary degree of complexity for the vast majority of use cases. There is a plan to potentially re-introduce this later and should anyone require it, please let us know.

Documentation

All of our documentation is designed for the cloud version of the product, however we'll be adding more detailed documentation for the self-hosted version soon. We've hidden away some features such as payment processing and SSL termination for custom domains that are not relevant to the self-hosted version.

The entire UI is powered by GraphQL using Apollo and the API documentation is automatically generated and easily navigated.

Support us!

All we ask is that you star or watch this on GitHub if you like the project. We'd love people to share and blog about this too! All issues will be responded to quickly in GitHub, and we're excited to see where this goes next.

Authors

Author	GitHub	LinkedIn
Christopher Beck
Alessandro Barzanti

scale8 / scale8-tag-manager-and-analytics

readme