search

HiromuHota / pdi-git-plugin

SpoonGit allows you to manage versions of local Kettle files without leaving Spoon. In addition to Git, Subversion is also supported.
Apache License 2.0
29 stars 12 forks source link

Add ability to Deploy git origin/master, merged/closed PR to server Enterprise Repo. #32

Open MatthewAlanCrouch opened 5 years ago

MatthewAlanCrouch commented 5 years ago

Currently Deploys to a remote Enterprise Repository require a 3rd party service listening for PRs or manually uploading files via User Console. Neither are necessarily very friendly to set up or maintain.

Pentaho (PDI) can itself get Deploy content using GitHub API's and upload to the Enterprise Repository with Pentaho Repository API's (our current preferred solution, along with GitHub Actions-Workflows). Adding this ability to Deploy a merged PR from spoongit would be a huge improvement on workflow efficiency

HiromuHota commented 5 years ago

Thanks for a feature request. In summary,

  1. You use GitHub (or alternatives) as a remote git repository, say origin, and use SpoonGit to push a branch feature/1 to this remote git repository.
  2. You use GitHub (or altenatives)'s web UI to create a PR (e.g., origin/feature/1 -> origin/master), make comments on it, merge/close it.
  3. You deploy all files in origin/master to a Pentaho Enterprise Repository currently using a 3rd party service or manual upload via Pentaho User Console.

You want SpoonGit to be able to do the deployment step.

I think you'd be just fine if SpoonGit can deploy a branch (e.g., origin/master) instead of a merged PR because this PR is already merged to a branch like origin/master. Correct me if I'm mistaken.

MatthewAlanCrouch commented 5 years ago
  1. Correct
  2. Correct
  3. Normally one would have to use a 3rd party integration or manually upload new merged PR’s to an Enterprise Repo. Or use this manual upload when any automation fails. We have a Pentaho job that gets the PR>commits> changed files>file content with the GitHub APIs and puts them on the Enterprise repo with the Pentaho APIs. Still requires some manual intervention but useable. We are also currently looking into the GitHub Actions where we can build a workflow to process similar to a webhook, but customized.

My thought would be that it would be a great additional feature for SpoonGit to have a simple deploy mechanism that would put the changed files to the Enterprise repo directly. The bigger issue that this really addresses, partially at least, is how to ensure that your local repo, the GitHub master (origin/master), and the remote Enterprise repo that the production server accesses all stays in sync. Thanks for you attention,

Matt Crouch Software Automation Developer [Tangoe]

From: Hiromu Hota notifications@github.com Sent: Wednesday, April 24, 2019 6:21 PM To: HiromuHota/pdi-git-plugin pdi-git-plugin@noreply.github.com Cc: Matt Crouch Matt.Crouch@tangoe.com; Author author@noreply.github.com Subject: [EXTERNAL]Re: [HiromuHota/pdi-git-plugin] Add ability to Deploy git origin/master, merged/closed PR to server Enterprise Repo. (#32)

Thanks for a feature request. In summary,

  1. You use GitHub (or alternatives) as a remote git repository, say origin, and use SpoonGit to push a branch feature/1 to this remote git repository.
  2. You use GitHub (or altenatives)'s web UI to create a PR (e.g., origin/feature/1 -> origin/master), make comments on it, merge/close it.
  3. You deploy all files in origin/master to a Pentaho Enterprise Repository currently using a 3rd party service or manual upload via Pentaho User Console.

You want SpoonGit to be able to do the deployment step.

I think you'd be just fine if SpoonGit can deploy a branch (e.g., origin/master) instead of a merged PR because this PR is already merged to a branch like origin/master. Correct me if I'm mistaken.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/HiromuHota/pdi-git-plugin/issues/32#issuecomment-486449682, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AD3WLEH2UK5IFBRHZXGK4XTPSDMMRANCNFSM4HIEOIHQ.

HiromuHota commented 5 years ago

I like the following idea

your local repo, the GitHub master (origin/master), and the remote Enterprise repo that the production server accesses all stays in sync.

but 1. it would be difficult to implement it and 2. it may not be appropriate to implement this feature in SpoonGit from architectural point of view.

  1. It'd be difficult to implement it

As you admitted it, your Pentaho job "still requires some manual intervention." I'd imagine that this Pentaho job assumes that nobody else but himself manipulates the Pentaho (Enterprise) Repository since the last sync, but when some unexpected things happen (e.g., somebody else manipulates the Repository) manual interventions are required. In order for this Pentaho job, or 3rd party script or whatever, to upload only the files modified/added/deleted by the last commit (could be trigger by a PR) without any need of manual intervention, he first has to figure out the status of the Pentaho Repository in the way that git does, which I don't think is straightforward.

One simple way of syncing I could imagine is to delete all files from the Pentaho Repository, no matter what status it is, and upload all files from the git repository. This is simple, reliable, but not efficient (especially when you have many files to upload).

  1. it may not be appropriate to implement this feature in SpoonGit from architectural point of view

I think this kind of sync would better be triggered by GitHub and executed by a server (instead of by a client like Spoon/SpoonGit). If this feature is implemented in SpoonGit, you still have to manually launch Spoon (and clicks "sync" button, depending on how it is implemented).

Please let me know if you have a different thought.

MatthewAlanCrouch commented 5 years ago

I think it could be designed and implemented without incredible difficulty, some error handling.

I had not thought of it’s architectural appropriateness in the way you describe:

“this kind of sync would better be triggered by GitHub and executed by a server (instead of by a client like Spoon/SpoonGit). If this feature is implemented in SpoonGit, you still have to manually launch Spoon (and clicks "sync" button, depending on how it is implemented).”

Yes, Absolutely better for a server to control it and not the client. This is actually the method I’m already exploring with GitHub Actions/Workflows triggered by the pull_request event.

Matt Crouch Software Automation Developer [Tangoe]

From: Hiromu Hota notifications@github.com Sent: Friday, April 26, 2019 1:48 PM To: HiromuHota/pdi-git-plugin pdi-git-plugin@noreply.github.com Cc: Matt Crouch Matt.Crouch@tangoe.com; Author author@noreply.github.com Subject: [EXTERNAL]Re: [HiromuHota/pdi-git-plugin] Add ability to Deploy git origin/master, merged/closed PR to server Enterprise Repo. (#32)

I like the following idea

your local repo, the GitHub master (origin/master), and the remote Enterprise repo that the production server accesses all stays in sync.

but 1. it would be difficult to implement it and 2. it may not be appropriate to implement this feature in SpoonGit from architectural point of view.

  1. It'd be difficult to implement it

As you admitted it, your Pentaho job "still requires some manual intervention." I'd imagine that this Pentaho job assumes that nobody else but himself manipulates the Pentaho (Enterprise) Repository since the last sync, but when some unexpected things happen (e.g., somebody else manipulates the Repository) manual interventions are required. In order for this Pentaho job, or 3rd party script or whatever, to upload only the files modified/added/deleted by the last commit (could be trigger by a PR) without any need of manual intervention, he first has to figure out the status of the Pentaho Repository in the way that git does, which I don't think is straightforward.

One simple way of syncing I could imagine is to delete all files from the Pentaho Repository, no matter what status it is, and upload all files from the git repository. This is simple, reliable, but not efficient (especially when you have many files to upload).

  1. it may not be appropriate to implement this feature in SpoonGit from architectural point of view

I think this kind of sync would better be triggered by GitHub and executed by a server (instead of by a client like Spoon/SpoonGit). If this feature is implemented in SpoonGit, you still have to manually launch Spoon (and clicks "sync" button, depending on how it is implemented).

Please let me know if you have a different thought.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub

HiromuHota commented 5 years ago

I'd implement this in a way that this can be executed in CLI instead of GUI. This could be an extension of this plugin,,, but wait, have you tried IvyGS - Ivy Git Sync? It says

IvyGS - Ivy Git Sysnc is a Sparkl plugin for help sinchronizer your BI solution that is storage on a git repository with Pentaho BI Server.

It is available on Pentaho Marketplace and can be installed to Pentaho Server, which sounds a perfect solution for your need.

HiromuHota commented 5 years ago

@latinojoel any thoughts?

latinojoel commented 5 years ago

Hi all,

IvyGS was developed a while back and the concept was proved by using this git PDI plugins: https://github.com/ivylabs/ivy-pdi-git-steps As this is built as a Sparkl plugin, it can be treated as a bootstrap tool to be adapted to the developer needs. It is possible to change the whole UI in CDE e and the backend in PDI.

Tagging @harrisward and @ivyis as they may be interested in that. Joel