search

adobe / helix-cli

Command-line tools for developing with AEM
Apache License 2.0
50 stars 61 forks source link

smoke tests fail - no idea how to investiate... #920

Closed tripodsan closed 5 years ago

tripodsan commented 5 years ago

the smoke tests are currently failing, I checked out project-helix.io, but on the local checkout, the tests pass.

@kptdobe can you teach us, how to investigate the failing smoke tests?

rofe commented 5 years ago

I created https://github.com/adobe/helix-continuous/issues/17 before seeing this one - feel free to duplicate as you see fit

tripodsan commented 5 years ago

ok, I tried the following:

  1. checkout helix-continuous
  2. clone helix-cli: 
    git clone --branch master --single-branch --depth 1 https://github.com/adobe/helix-cli.git
  3. setup gdm 
    cd scripts/gdm; npm install; cd ..
  4. run gdm (would be nice it if would show a summary of the used modues) 
    cd helix-cli; node ../scripts/gdm/index.js; cd ..
  5. clone project-helix.io and install: 
    git clone --single-branch --depth 1 -b master https://github.com/adobe/project-helix.io.git
cd project-helix.io; npm install; cd ..
  6. run the smoke tests: 
    cd project-helix.io; env HLX_SMOKE_EXEC='node ../helix-cli/index.js --log-level warn' npx mocha --exit test/smoke/*
Running smoke test using: node ../helix-cli/index.js --log-level warn

fails!

tripodsan commented 5 years ago

it seems to be an issue with the attribute value output in the htl engine:

            <!-- include ${content.nav}.summary.html-->
            <xesi:include src="${content.nav}.summary.html" />

produces:

            <!-- include &#x2f;SUMMARY.summary.html-->
            <xesi:include src=".summary.html" />
tripodsan commented 5 years ago

https://github.com/adobe/htlengine/issues/64

tripodsan commented 5 years ago

@kptdobe once problem of the smoke tests is, that GDM uses the master of all @adobe dependencies. even if not referenced in the dependencies of the tested project.

eg:

htlengine is a dependency of parcel-plugin-htl but not of helix-cli, but GMD pulls the latest, even if parcel-plugin-htl is not smoke-tested:

during smoke test:

$ npm ls @adobe/htlengine
@adobe/helix-cli@3.2.2 helix-continuous/helix-cli
├── @adobe/htlengine@2.3.0  (github:adobe/htlengine#b8875c637d629620d4bac60751fa2d7aaf2d3141)
└─┬ @adobe/parcel-plugin-htl@2.1.9 (github:adobe/parcel-plugin-htl#73f87d97083361a46a082cc8dc259eb58a5664d9)
  └── UNMET DEPENDENCY @adobe/htlengine@2.2.2

in normal helix-cli:

$ npm ls @adobe/htlengine
@adobe/helix-cli@3.2.2 helix-cli
└─┬ @adobe/parcel-plugin-htl@2.1.9
  └── @adobe/htlengine@2.2.2
tripodsan commented 5 years ago

imminent problem is solved. keeping open for documentation.

kptdobe commented 5 years ago

hummm.. I would expect that the htlengine dependency gets update to master code also in the parcel-plugin-htl dep: the DGM "should" link sub dependencies of dependencies - see https://github.com/adobe/helix-continuous/blob/master/scripts/gdm/index.js#L149-L170.

The cli that gets created by the GDM is "master" related not "npm version" related: it takes the master branch from all projects, completely independently from npm versions. Goal was to make sure that whatever goes into master from one of our dependencies do not break the cli. If you release 20 versions of htlengine but only make a PR to upgrade from v2 to v20 in the cli, you do not know which intermediate version broke the cli. With the continuous release we have now, we could simplify this and only smoke test against last cli master and branch from module being changed (+ sub dependencies update).

I am documenting now the "investigation" process on helix-continuous README.

kptdobe commented 5 years ago

I documented the first level investigation process here: https://github.com/adobe/helix-continuous/pull/18 Direct access to doc section: https://github.com/adobe/helix-continuous/blob/063de67f54fc0f750766cc283c67b862b045cbe0/.circleci/orbs/helix-smoke-tests/README.md#debug

tripodsan commented 5 years ago

. I would expect that the htlengine dependency gets update to master code also in the parcel-plugin-htl dep

not always...

for example, assume a transitive dependency (like htlengine, or git-server) has a breaking change, and is deliberately not updated in the direct dependency.

In any case, I don't want to use the master of all modules, just because I smoke test my changes in made in 1 module.

kptdobe commented 5 years ago

for example, assume a transitive dependency (like htlengine, or git-server) has a breaking change, and is deliberately not updated in the direct dependency.

The pb comes for the next person coming right after, just patching htlengine or git-server for example and wanting to update the direct dependency... He will have failing smoke tests while he does not introduce a breaking change.

The core issue is that master is the source of truth for all repos, introducing a breaking change, merge and not updating the direct dependency cannot work as is for now.

The current model prevent from that which is, I think, the most annoying thing. I am not saying it is perfect, I agree that we might need to iterate or do differently and maybe make it better.