dataplat / dbops

⚙ dbops - Powershell module that provides continuous database deployments on any scale
MIT License
163 stars 39 forks source link

Capability to detect precedence in scripts? #119

Open jeffchulg opened 4 years ago

jeffchulg commented 4 years ago

Hi,

I've just learned about this module and I would like to know if there is a way to detect dynamically scripts dependencies so that the overall execution will be successful?

I explain myself: I'm used to create one script per database object. This script is re-runnable (it checks if the object exists or one of its column/index exists and if not => it adds it). In most of cases, There is a hierarchy between scripts. For instance, the script running the CREATE SCHEMA [a] should be run before any script that will create an object in that schema.

Thanks in advance for the explanation.

nvarscar commented 4 years ago

Currently, there is no capability of detecting dependancies in the module. And it would be quite an undertaking to implement something like this. The only way of approaching this that I can see is to implement a language parser that would distinct object dependencies and build them up in a dependency chain. Then it would be possible to rename the files in a proper order. Not to mention, it becomes significantly more complex to have this functionality across different platforms.

jeffchulg commented 4 years ago

Well, I've been using Combiner for more than 6 years, even if it's been designed for CSS combination in the first place...

It's written in java, so cross-platform and you only need to add "/*requires <filename>*/" at the beginning of your files.

I don't know why, but it's sometime tricky to get the dependency tree match your need if you have a lot of files, but if you get used to develop "feature-based" or "plugin-based" it works! (the divide and conquer principle)

Hope this helps !

jeffchulg commented 4 years ago

Here is the code that wraps up combiner:

function Build-File {
    param (
        [string]$BUILT_FILE,
        [string]$BUILD_LOG,
        [string[]]$SRC_STR
    )
    Write-Host "Now building ${BUILT_FILE}"

    java -jar $CombinerToolPath --charset UTF8 -v -e -o ${BUILT_FILE} ${SRC_STR} 2>&1 | Where-Object {$_ -match '\S'} | Out-File -Encoding UTF8 "${BUILD_LOG}" -width 250

    $nbErrors = Select-String -pattern "^\[ERROR\]" -path "${BUILD_LOG}"

    $nbErrors = Select-String -pattern "^\[ERROR\]" -path "${BUILD_LOG}"
    $nbFilesNotFound = Select-String -pattern "Couldn't find file" -path "${BUILD_LOG}"

    if($nbErrors -ne $null) {
        Write-Host "    > ERROR - Check Log ${BUILD_LOG}"
        foreach ($err in $nbErrors) {
            Write-Host "         At line $($err.LineNumber)"
        }
        throw "An error occurred during combiner execution"
    }

    if($nbFilesNotFound -ne $null) {
        Write-Host "    > ERROR - Check Log ${BUILD_LOG}"
        foreach ($err in $nbFilesNotFound) {
            Write-Host "         At line $($err.LineNumber)"
        }
        throw "An error occurred before combiner execution"
    }
    return $true
}
nvarscar commented 4 years ago

So, if I understand correctly, this solution still requires manually added requirements "tag" in the file, and then combines them all in one single file in the right order. That's not exactly a fully automatic precedence detection and you can pretty much achieve the same thing by renaming files to be in a proper precedence order.

jeffchulg commented 4 years ago

it's nearly correct...

Except that if you number the files and a change comes "between" two consecutive numbers, or involve a complete review or the sequence, then, with this way, you don't need to bother with that, it will be performed dynamically.