They had a data platform, serve analytics, ML teams, data they process, ingest from several sources, run it through Spark, goes out to Hive tables, Redshift, more recently switched over to a new platform that's based on databricks.
going through databricks delta tables
involved in several projects with the original paltform, optimizing some of the pipelines, security and encryption, was rotating all of the encryption keys they were using to a new set of keys,
doing a lot of data cleaning, build an automated system to clean a users private information from the data lake
tech stack:
Huge re-org
data platform got retired
Recently moved to a new team, not related to Data
Would be open to moving back to Data Engineering
actively look for a new position?
yes
looking for?
hoping to find a good stable company
how do you define stable?
well established, product market fit, defensible position in the market
https://www.linkedin.com/in/ted-y-255b563
Scope of role:
tech stack:
Huge re-org
actively look for a new position?
looking for?
how do you define stable?
compensation:
location
time frame?