Study where Firebase Emulators take their time...

akauppi / firebase-jest-testing

Firebase back-end testing, using Jest.

MIT License

21 stars 3 forks source link

Study where Firebase Emulators take their time... #25

Open akauppi opened 3 years ago

akauppi commented 3 years ago

This would be useful for:

regression testing, especially when firebase-tools or firebase-admin versions change
showing if there are unnecessarily slow behaviours from the Firebase Emulators, so Firebase could optimise these

For example: it looks that the first watch for a Firestore collection takes 300ms extra. Where is this time spent? Is it really necessary, for emulators that run in memory? Once we can show statistics, the ball can be rolled to Firebase's side of the field, so they would make the overall test execution yet speedier.

akauppi commented 3 years ago

Based on napkin notes, the slow starts only occur on Docker Compose on macOS (local CI-like development).

Since this is not one of the actual use cases of the repo, this can be on low flame. However, it does point to Firebase Emulators having performance issues when run under Docker Desktop for Mac, and should therefore be studied.

Anything < 2000ms per test is regarded adequate (CI is close to this, and may sometimes exceed it).

Napkin:

fns slowest (ms)	rules slowest (ms)
DC on macOS	2655, 2569	2414, 2304
DC on Windows	1679, 1700, 2013	dragons
CI - Cloud Build	1925	1200, 1218
Native macOS	1230, 1187	660, 667
Native Windows+WSL2	1382, 1466	1081, 1052

akauppi commented 3 years ago

Managed to bring the execution times down, by #37

Tracing code is left in exp-timings branch (not merged). It didn't reveal anything vital, yet. My gut feel is that actual optimisations need looking into the internals of how Firebase Emulators are run. For now, since CI issue is solved, there's less need for that.

akauppi commented 2 years ago

Status:

I would be extremely happy 🤪 if someone has the time to do this. Unfortunately, I don't. Here are the current (12-Mar-2022) timings as a baseline.

$ cd sample
$ time npm test
real    0m20.982s
user    0m10.525s
sys 0m1.663s

# round 2...
real    0m16.940s
user    0m10.251s
sys 0m1.476s

# once more..
real    0m16.866s
user    0m10.261s
sys 0m1.518s

$ cd sample.dc
# first time omitted - starts the emulators

$ time npm test
real    0m18.883s
user    0m10.133s
sys 0m1.564s

#... 2nd
real    0m13.504s
user    0m9.507s
sys 0m1.427s

#... 3rd
real    0m14.641s
user    0m9.701s
sys 0m1.438s

I'd like to get the times down to 5s, ideally.

Both on Mac Mini 2018 (Intel), 4 cores, 16MB, macOS 12.2. Docker Desktop for Mac 4.5, containers given 2GB RAM, 1GB swap, 2 cores

akauppi commented 2 years ago

Just a note that: with Jest 28.0.0-alpha.8 and firebase-tools 10.6.0, the timings are similar.

launching emulators (first npm test in sample.dc) takes 45 s

Note:

With Jest 28.0.0-alpha.8 and firebase-tools 10.6.0, functions tests usually take > 2000ms so run them like so:

$ WARM_UP_TIMEOUT=6000 time npm test
...
 PASS  ../sample/test-fns/greet.test.js
  Cloud Function callables
    ✓ returns a greeting (4028 ms)

 PASS  ../sample/test-fns/userInfo.test.js
  userInfo shadowing
    ✓ Central user information is distributed to a project where the user is a member (3625 ms)
    ✓ Central user information is not distributed to a project where the user is not a member (313 ms)
...

This only proves the need to study where such time are spent.