Tests fail on all recent PRs

smile13241324 commented 6 years ago

Description :octocat:

I have changed some details on the "org" mode documentation and noticed that some tests do fail consistently for all recent PRs.

Namely these are: FAILED test-configure-packages-2--excluded-package-is-not-configured FAILED test-configure-packages-2--lazy-install-package-is-not-configured FAILED test-configure-packages-2--local-package-w/-bad-string-location-gives-warning FAILED test-configure-packages-2--local-package-w/-dotfile-owner-update-load-path FAILED test-configure-packages-2--local-package-w/-layer-owner-update-load-path FAILED test-configure-packages-2--local-package-w/-string-location-update-load-path FAILED test-configure-packages-2--local-package-w/o-owner-doesnt-update-load-path FAILED test-configure-packages-2--package-owned-by-dotfile-is-not-configured FAILED test-configure-packages-2--package-w/-layer-owner-is-configured FAILED test-configure-packages-2--package-w/o-owner-is-not-configured FAILED test-configure-packages-2--protected-excluded-package-is-configured FAILED test-configure-packages-2--protected-package-is-configured FAILED test-configure-packages-2--site-package-is-configured FAILED test-configure-packages-2--toggle-nil-is-not-configured FAILED test-configure-packages-2--toggle-t-is-configured

Reproduction guide :beetle:

Start Emacs
Run the tests or open a PR
Notice that 15 tests fail on all Emacs versions

Observed behaviour: :eyes: :broken_heart: Tests fail on all recent branches

Expected behaviour: :heart: :smile: Tests only fail if there is an issue with the contribution

System Info :computer:

OS: gnu/linux
Emacs: 26.1
Spacemacs: 0.300.0
Spacemacs branch: develop (rev. b54ea8ef9)
Graphic display: t
Distribution: spacemacs
Editing style: vim
Completion: helm

Layers:

((helm :variables helm-enable-auto-resize t)
nim ocaml scheme
(auto-completion :variables auto-completion-return-key-behavior 'complete auto-completion-tab-key-behavior 'cycle auto-completion-complete-with-key-sequence nil auto-completion-complete-with-key-sequence-delay 0.0 auto-completion-private-snippets-directory nil auto-completion-enable-snippets-in-popup nil auto-completion-enable-help-tooltip t auto-completion-enable-sort-by-usage t)
better-defaults
(clojure :variables clojure-enable-fancify-symbols t)
gpu graphviz
(plantuml :variables plantuml-jar-path "~/.plantuml/plantUml.jar" org-plantuml-jar-path "~/.plantuml/plantUml.jar")
salt ibuffer emacs-lisp pdf cscope gtags
(org :variables org-want-todo-bindings t org-enable-github-support t org-enable-bootstrap-support t org-enable-reveal-js-support t org-enable-hugo-support t)
(shell :variables shell-enable-smart-eshell t shell-default-shell 'ansi-term shell-default-height 30 shell-default-position 'bottom shell-default-full-span nil shell-default-term-shell "/usr/bin/fish" syntax-checking)
version-control semantic templates themes-megapack
(c-c++ :variables c-c++-default-mode-for-headers 'c++-mode c-c++-enable-clang-support t c-c++-enable-clang-format-on-save t)
(cmake :variables cmake-enable-cmake-ide-support t)
git github sml pass common-lisp
(python :variables python-backend 'lsp python-test-runner 'pytest python-enable-yapf-format-on-save t python-sort-imports-on-save t)
windows-scripts agda jr shell-scripts prodigy emoji
(markdown :variables markdown-live-preview-engine 'vmd)
bibtex json debug
(latex :variables latex-enable-auto-fill t latex-enable-magic nil latex-enable-folding t)
lua html
(javascript :variables javascript-disable-tern-port-files nil)
react yaml web-beautify nginx docker csv search-engine
(restclient :variables restclient-use-org t)
restructuredtext selectric octave purescript speed-reading systemd imenu-list
(java :variables java-backend 'meghanada)
groovy
(go :variables go-use-gometalinter t gofmt-command "goimports" go-tab-width 4 go-use-gocheck-for-testing t go-use-test-args "-race -timeout 10s" godoc-at-point-function 'godoc-gogetdoc)
coq django
(elm :variables elm-sort-imports-on-save t elm-format-on-save t)
idris
(haskell :variables haskell-enable-hindent-style "fundamental" haskell-completion-backend 'dante haskell-enable-hindent t)
pandoc sphinx lsp parinfer asciidoc neotree ansible puppet rebox rust xkcd typography vimscript
(terraform :variables terraform-auto-format-on-save t)
(ruby :variables ruby-enable-enh-ruby-mode t ruby-version-manager 'rvm)
ruby-on-rails asm
(sql :variables sql-capitalize-keywords t)
perl6 autohotkey
(elixir :variables flycheck-elixir-credo-strict t)
faust vagrant erlang
(dash :variables helm-dash-docset-newpath "~/.local/share/Zeal/Zeal/docsets")
games php nginx racket
(colors :variables colors-colorize-identifiers 'all)
d
(typescript :variables typescript-fmt-tool 'typescript-formatter typescript-fmt-on-save t))

System configuration features: XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 MODULES THREADS LIBSYSTEMD LCMS2

sdwolfz commented 6 years ago

I'll take a look and see if I can figure it out.

sdwolfz commented 6 years ago

Pushed 25827266a71be946d760b9d015ffff37563162f7 to disable the broken assertions but I don't know how to fix it properly, waiting for @syl20bnr to continue his work in that area.

dalanicolai commented 4 years ago

It appears to me that some tests still fail consistently, i.e. the elisp tests / test (windows-latest, 26.3, core) (pull_request) fails in the latest eight pull requests (starting from #13907).

Now I wonder if these PR's are still candidates/evaluated for merging (???). I have created #13927, and I see no reason why it should have broken anything.

smile13241324 commented 4 years ago

@dalanicolai I don't think that your PR is causing these problems in CI and we are seeing the two mentioned PRs as candidates for merging. However #13907 is a draft release, therefore I have not checked it. I have read through #13927 already though I am not yet totally clear about what behaviour you have changed.

I don't know why the mentioned test fails, but I think @JAremko knows about these already.

JAremko commented 4 years ago

Yeah we added more tests and found more bugs... I'm thinking about disabling some tests in PR testing and such, for now. But it kinda stinks. :thinking: windows-latest, 26.3, core seems to be busted in spacelpa department and @syl20bnr planned to work on it.

dalanicolai commented 3 years ago

Thanks for the replies!

@smile13241324 I will elaborate on the (small but very useful) changes in #13927 in the conversation of the PR itself.

syl20bnr / spacemacs