search

mdrakiburrahman / rakirahman.me

💻Personal blog powered by Gatsby
https://www.rakirahman.me
MIT License
1 stars 1 forks source link

Purview SHIR set-up including JDK and Drivers #5

Closed tfayyaz closed 3 years ago

tfayyaz commented 3 years ago

Hi,

Thanks so much for the great post on automating the set-up for Purview SHIR.

I was wondering if you were able to automate installing and downloading things like JDK 11, the drivers needed for Hive Metastore and BigQuery and SSL certificates?

If you do have examples of how you achieved this I would love to take a look.

If not I will share back here anything else I find.

Thanks Tahir

mdrakiburrahman commented 3 years ago

Hi @tfayyaz,

Thanks for the note, glad you found it useful!

So I haven't spent any time on installing JDK/BQ pre-reqs etc (since the problem I was trying to solve was SHIR lifecycle automation without the az CLI being out yet) - but let me offer some info you might find helpful.

  1. ARM Custom Script Extension: If you're deploying the SHIR VM in Azure, I would recommend leveraging ARM Custom Script Extension.

    So basically what I would do, is in the ARM Template that deploys the SHIR VM from scratch, point it to a Powershell script that contains your SHIR Install > JDK 11 download, BQ specific artifact bootstrap logic etc. So what'll happen, is as the VM is being deployed, your script will install everything the VM needs (JDK, BQ stuff etc.). When the deployment completes, the VM is ready to go. If you ever need to rotate the VM, just reuse the logic in the blog post, and redo the ARM deployment.

    Example: Sample Custom Script call in ARM - this example isn't SHIR specific, another repo I'm working on, Sample of Powershell script you would point the extension to (this is specific to SHIR - you can reuse this for BQ pre-reqs).

    The PowerShell gist I linked above has SHIR download and install logic. You should be able to extend this script if you can point it to the JDK download link. Better yet, if you store these artifacts in a Blob Storage and generate SAS URLs, or something like JFrog, it'll help the SHIR pull consistently without Internet access/links breaking. Then, just install the binary on the VM via PowerShell (msiexec or some other Powershell command - based on your binary).

  2. Terraform and WinRM: I'm assuming this will involve GCP since you're pulling from BQ (so it'll be a GCP SHIR VM) - so the ARM Custom Script above won't be relevant. So you can extend the same PowerShell gist I shared and modify it with JDK/BQ pre-req logic, and use Terraform file provisioner to run the script before the VM spins up on GCP - here's an example. Note that WinRM (which TF file provisioner uses) is finicky, I found ARM much easier/more reliable to use on Azure, since it injects the Script using the Extension built into ARM (whereas with WinRM you're using Windows native stuff).

So in either scenarios, you can use ARM or TF to bootstrap the VM to execute Custom Logic (Powershell), and your actual uplift task would be to extend the Powershell script so that it captures you're pre-reqs.

Hope that helps - if you have any q's please comment here. If you also build something for this, I'd love to learn about it if you'd like to share 😊

tfayyaz commented 3 years ago

Thank you for detailed response and examples. I will look into all of those and let you know how I get on.

Thanks Tahir