Scaling and profiling experiments

lsst-uk / lasair-project-management

Event handling site for LSST:UK

Apache License 2.0

9 stars 0 forks source link

Open tms-epcc opened 2 years ago

tms-epcc commented 2 years ago

From cycle-6 wiki page this comprises:

[x] get it working first with acceptance tests for Lasair 4.0 then enhance as follows:
[ ] See Running pipeline at speed, scaling experiments below
[ ] Numbers of nodes, processes, threads
[ ] Batch and buffer sizes
[ ] Database tuning
[ ] LSST rates
[ ] process 10 million alerts in 12 hours
[ ] Trans atlantic link (blocked)
[ ] Maximising the pipeline throughput and the web/api response

tms-epcc commented 1 year ago

05/OCT/22

@RoyWilliams reported development system works with Lasair v4.0 acceptance tests
@tms-epcc and @RoyWilliams agreed to change ticket name from "Build complete development system" to "Scaling and profiling experiments"

tms-epcc commented 1 year ago

06/OCT/22

Cycle 7 plan added the following tests to the task and hence the check box list above
- Trans atlantic link (blocked)
- Maximising the pipeline throughput and the web/api response

tms-epcc commented 1 year ago

14/DEC/22

@gpfrancis will provide initial thoughts to promote discussion on the test plan for this

tms-epcc commented 1 year ago

25/JAN/23

@RoyWilliams reported that the Performance Testing Plan was now ready and the next step was to agree responsibilities for executing the various tests

tms-epcc commented 1 year ago

16/AUG/23

as discussed with @RoyWilliams this will be deferred to cycle 9 (starts 01/OCT/23) due to the impact of RSP synergy, Oxford user meeting and LSST@Europe

tms-epcc commented 4 weeks ago

14/AUG/24

10 million alerts in 48 hours was achieved as part of the OR4 : see July'24 newsletter