Load test: sending to lobid-resources subject inboxes

hbz / lobid-resources

Transformation, web frontend, and API for the hbz catalog as LOD

http://lobid.org/resources

Eclipse Public License 2.0

7 stars 7 forks source link

Load test: sending to lobid-resources subject inboxes #986

Closed fsteeg closed 4 years ago

fsteeg commented 5 years ago

Every day, for every new lobid-resource, for each of its subjects, POST an LDN to the subject's inbox, e.g. to https://test.skohub.io/inbox?target=http://test.lobid.org/gnd/4020517-4
Send the lobid-resource's full JSON as the POST body
Spread the updates over the day to simulate live updates

While we don't have a live system behind lobid-resources, the idea here is to simulate the load created by multiple senders. To increase the amount of messages, we could change this from subjects to authors in the future.

dr0i commented 5 years ago

Wrote a gist https://gist.github.com/dr0i/72fdc385fc4e72a6930f1644523bbdc6. When https://test.skohub.io/ inbox is up, it may be crontabbed.

acka47 commented 5 years ago

When https://test.skohub.io/ inbox is up, it may be crontabbe

https://test.skohub.io/ is up now, see https://github.com/hbz/skohub-pubsub/pull/12

dr0i commented 5 years ago

Up'n running. Crontabbed starting every day at 09:00.

fsteeg commented 5 years ago

Just a heads-up for @literarymachine: https://test.skohub.io/ should start receiving automated traffic starting tomorrow 09:00, see https://github.com/hbz/lobid-resources/issues/986#issue-429332959 (first comment in this issue).

dr0i commented 5 years ago

@literarymachine @fsteeg it's already started and sending. And will , from now on, start every day at 9:00.

dr0i commented 5 years ago

Create #992 to be able to see what's happening. @fsteeg If this issue is good you can close it.

acka47 commented 5 years ago

Talking to @literarymachine at ELAG2019, he said that it would make sense to make the load test a bit harder by sending out all the notifications at once as soon as the daily updates are in. Reopening to test this scenario for some days.

dr0i commented 5 years ago

Removed the sleep. This script will now work as fast as it can, I expect no more than 10/s.

literarymachine commented 5 years ago

Removed the sleep. This script will now work as fast as it can, I expect no more than 10/s.

I haven't closely monitored the server, but it is still up and running. Thinking about this, I should probably add some logging: https://github.com/hbz/skohub-pubsub/issues/14

literarymachine commented 5 years ago

To increase the amount of messages, we could change this from subjects to authors in the future.

Why not both?

dr0i commented 5 years ago

Is this running test actually useful or can I stop if it's indeed unnecessarily producing load?

acka47 commented 5 years ago

I guess you can stop it for now.

acka47 commented 4 years ago

@dr0i have you stopped the load test already? I not, please do and close this issue when it is stopped.

dr0i commented 4 years ago

Yeah I stopped it already at 11th July. For reference, it can be started again enabling dispatchLobidResourcesToSkohubInbox.sh, see crontab. Closing.