search

istanbuljs / nyc

the Istanbul command line interface
https://istanbul.js.org/
ISC License
5.54k stars 353 forks source link

nyc generates no output on TravisCI #993

Open chrisveness opened 5 years ago

chrisveness commented 5 years ago

Link to bug demonstration repository.

Expected Behavior

image

Observed Behavior

image

.travis.yml runs after_success: nyc npm test && nyc report --reporter=text-lcov | coveralls which consistently fails as shown above.

I think have inconsistently seen a similar issue in my local environment, but haven't been able to reproduce it consistently.

Forensic Information

Ubuntu 14.04.5 LTS, Mac OS X 10.13.3, Windows Node 8.15.0, 10.15.1, 11.10.0, nyc 13.3.0

coreyfarrell commented 5 years ago

Could you try setting the ESM_DISABLE_CACHE environmental variable noted in https://github.com/standard-things/esm? Maybe try cross-env ESM_DISABLE_CACHE=1 nyc --cache=false npm test? Just to try ruling out issues caused by caching.

chrisveness commented 5 years ago

Thanks for the suggestion.

However, I changed the after_success step to

after_success:
- ESM_DISABLE_CACHE=1 nyc --cache=false npm test && nyc report --reporter=text-lcov | coveralls

and it's made no difference https://travis-ci.org/chrisveness/geodesy/jobs/494203195#L2780 :(

MrSwitch commented 5 years ago

Downgrade to 13.2.0 - all was well after that 😢

coreyfarrell commented 5 years ago

I just tried this locally, 1 out of 10+ runs I got failed coverage. I'm not sure the cause (it's difficult to troubleshoot something that happens so inconsistently). The failure I got was with nyc@13.2.0 so downgrading did not fix the issue. I know we've had a number of issues interoperating with esm, I'm not sure those are all fully resolved.

@MrSwitch if you are having issues that are fixed by downgrading to nyc@13.2.0 please open a ticket with the details.

srepollock commented 5 years ago

There was also an issue with Coveralls last night. That's why you were getting a 422 error. Seems to have been fixed up now.

EDIT: After reading the thread again I realize I'm completely off base. Apologies. I have also been seeing this error in my own nyc output too, however.

chrisveness commented 5 years ago

@MrSwitch I tried downgrading to nyc 13.2.0 and it has unfortunately made no difference.

@coreyfarrell I have for some time had odd behaviour with nyc; mostly when I first ran coverage after doing some other development work. I found that if I ran nyc a second time, it would be fine, so that became my workaround. I cannot find any way of reliably reproducing this issue on my development machine (Linux Mint 18) – it seems a mystery what might be different on the TravisCI VMs. I have noticed that nyc generates 3 files in .nyc_output, two of which contain nothing but ‘{}’, and wondered whether that might be in some way connected, but I haven’t had time to delve into the workings of nyc.

chrisveness commented 5 years ago

@coreyfarrell following up on the .nyc_output files, I thought I would append ls -l .nyc_output to the after_success step: it turns out there are 3 files of 2 bytes each – I don't know if that might suggest anything to you?

image

coreyfarrell commented 5 years ago

The {} content is supposed to happen when a process does not load any source that should be covered. So when you run nyc npm test this is supposed to produce {} for the npm process since that process does not load any sources that are to have coverage. I'm unsure where the second {} comes from, if npm uses 2 process spawns or if mocha has a parent and worker processes (maybe to detect if the worker process crashes).

The 3rd empty {} is our problem, I'm also unsure what could be causing that.

thePunderWoman commented 5 years ago

I'm seeing the exact same behavior in circleci. I get full coverage locally, but in CI I see all the .nyc_output json files as a totally empty object and coverage shows no coverage at all. I've been messing with it all day so far to no avail. I'm going to attempt the downgrade to 13.2.0 to see if it works.

thePunderWoman commented 5 years ago

Even downgrading, I see no coverage results in CI. So I don't think we're seeing the same issue. I'll have to keep digging. I'll watch this issue to see if anything comes of it.

srepollock commented 5 years ago

Is there not a way to attach nyc to the mocha process?

chrisveness commented 5 years ago

I have found a workaround (not a fix).

Remembering that a second run of nyc would fix the intermittant problems I had in my development environment, I added a second run to the Travis CI config: my .travis.yml now includes

after_success:
- nyc npm test
- nyc npm test && nyc report --reporter=text-lcov | coveralls

and correct stats are now sent to coveralls.

Whether that helps you in locating the actual issue, I don't know!

thePunderWoman commented 5 years ago

I figured out what my issue was. On CI, I forgot to set NODE_ENV to test. I managed to stumble upon this when a team member had it happen locally.

chrisveness commented 5 years ago

@janiukjf I think we must as you say be seeing different issues; I tried modifying my .travis.yml to

env:
- NODE_ENV=test

after_success:
- nyc npm test && nyc report --reporter=text-lcov | coveralls

but I simply got my original observed behaviour :(

tomas2387 commented 5 years ago

I saw the same behaviour: all coverage to zero and .nyc_output empty.
I installed the last version of node and everything worked again.

I'm trying to go back to the buggy version with nvm, but until now I can't reproduce the buggy behaviour. It's really weird.

OS: MacOS Mojave

Buggy node/npm version (as I remember):

Working after node update:

chrisveness commented 5 years ago

I think this problem is unchanged.

It is less urgent for me as I found the workaround (above) of running nyc npm test twice.

I still have similar problems in my local development environment, and sometimes don't seem to be able to generate coverage at all.