search

openbaton / NFVO

Repository containing the source code of the NFVO
Apache License 2.0
61 stars 52 forks source link

"Deadlock found when trying to get lock; try restarting transaction" when instantiating or deleting NSR #140

Closed tomasbrn closed 7 years ago

tomasbrn commented 7 years ago

Sometimes when instantiating a new NSR or deleting an existing NSR, the following error occurs:

2017-04-19 15:45:54.710  WARN 8043 --- [OpenbatonTask-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 1213, SQLState: 40001
2017-04-19 15:45:54.711 ERROR 8043 --- [OpenbatonTask-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : Deadlock found when trying to get lock; try restarting transaction
2017-04-19 15:45:54.712  WARN 8043 --- [OpenbatonTask-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Warning Code: 1213, SQLState: 41000
2017-04-19 15:45:54.713  WARN 8043 --- [OpenbatonTask-3] o.h.engine.jdbc.spi.SqlExceptionHelper   : Deadlock found when trying to get lock; try restarting transaction
2017-04-19 15:45:54.726 ERROR 8043 --- [OpenbatonTask-3] o.o.n.v.tasks.abstracts.AbstractTask     : There was an uncaught exception. Message is: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement

This puts the NSR in an ERROR state; when deleting this also fails to remove the NSR (all the resources etc. are removed though, only the record remains).

The NSD I am using contains 4 VNF packages, A, B, C and D. A, B and C all have a dependency towards D. C have dependencies towards A and B.

I'm not doing anything special for this to occur, just 1) Create a new NSD with the 4 packages 2) Launch the NSD

so maybe it has something do to with the number of VNF packages and the dependencies between them.

I've uploaded stripped versions of our vnf packages, in case they can be of any help.

edit: well I tried to upload to github but it refused, here's a dropbox link: https://www.dropbox.com/s/2buy0p7epktld0t/nfv_packages.zip?dl=0

marcellom commented 7 years ago

Hi, thanks for the detailed explanation. I managed to replicate this bug, however finding the cause is very tricky. The deadlock fires here: https://github.com/openbaton/NFVO/blob/develop/vnfm/vnfm-impl/src/main/java/org/openbaton/nfvo/vnfm_reg/tasks/abstracts/AbstractTask.java#L108. And the last log, before this exception, is here https://github.com/openbaton/NFVO/blob/master/vnfm/vnfm-impl/src/main/java/org/openbaton/nfvo/vnfm_reg/VnfmManager.java#L600. I suppose a deadlock between two threads that wait each other. I forward this issue to @lorenzotomasini which implemented the above methods.

gc4rella commented 7 years ago

still open? should be checked for upcoming release.

lorenzotomasini commented 7 years ago

hi @tomasbrn i just tried your example, adapting the image and vim instance names and everything went smooth at first try. probably there were some unknown fix 😄 in the develop branch and now works somehow.

here the logs generic_logs.txt nfvo_logs.txt plugin_logs.txt