Failed to acquire write lock for id: -333

[renaldas]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

11.3.1

Bug summary

When trying to save node getting this error, it happens on random nodes, sometimes you need to hit 3 times to publish the page.

Site runs on Azure with Azure SQL, all configuration set as per recommendations for Azure and also just tried to save and publish page locally, same thing happens.

Specifics

No response

Steps to reproduce

Expected result / actual result

No response

[github-actions[bot]]

Hi there @renaldas!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face:

[Zeegaan]

@renaldas What configuration are you running with? I know you said as per recommendations, but are those from the docs or ? 🤔 (Regardless, I'd love a link 👍)

[renaldas]

@Zeegaan

Config based on: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps

I also increased: DistributedLockingReadLockDefaultTimeout and DistributedLockingWriteLockDefaultTimeout to max

"CMS": {
      "ModelsBuilder": {
        "ModelsMode": "Nothing"
      },
      "Global": {
        "UseHttps": true,
        "MainDomLock": "FileSystemMainDomLock",
        "DistributedLockingReadLockDefaultTimeout": "00:02:00",
        "DistributedLockingWriteLockDefaultTimeout": "00:00:20"
      },
      "Hosting": {
        "LocalTempStorageLocation": "EnvironmentTemp"
      },
      "RuntimeMinification": {
        "UseInMemoryCache": true,
        "CacheBuster": "Version",
        "Version": "20230504"
      },
      "Examine": {
        "LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory"
      },
      "Content": {
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": false
        },
      }    
    }

[tnielsenskybruddk]

I experience the same problem in a 10.4.2 solution. Also running on Azure SQL.

My configuration looks like this:

    "CMS": {
      "Runtime": {
        "MaxRequestLength": 40000
      },
      "Global": {
        "SanitizeTinyMce": true,
        "ReservedUrls": "~/sitemap.xml",
        "ReservedPaths": "~/api/,~/swagger",
        "UseHttps": true,
        "TimeOut": "02:00:00",
        "Smtp": {
          "From": "",
          "Host": "",
          "Port": 25
        }
      },
      "RuntimeMinification": {
        "UseInMemoryCache": false,
        "CacheBuster": "Version",
        "Version": "49"
      },
      "Content": {
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": true
        }
      },
      "ModelsBuilder": {
        "ModelsMode": "Nothing",
        "ModelsDirectory": "~/../code/Models/Umbraco",
        "ModelsNamespace": "Code.Models.Umbraco",
        "UseDirectories": true,
        "AcceptUnsafeModelsDirectory": true
      },
      "RichTextEditor": {
        "CustomConfig": {
          "advlist_bullet_styles": "default",
          "advlist_number_styles": "standard,upper-alpha"
        }
      },
      "Security": {
        "AllowEditInvariantFromNonDefault": "true"
      }
    },

[Zeegaan]

@tnielsenskybruddk is this something happening to you frequently? And how frequent?🤔

[tnielsenskybruddk]

@Zeegaan That is very diffent. Last week it came once in a while. At the moment it comes all the time.

I first experienced the problem on April 27, 2023.

[bjarnef]

@Zeegaan I am frequently seeing this on an Umbraco v10.4.0 project. Often when saving content types.

I often get this error when working locally.

Waiting a few minutes the document type is saved without errors.

[bjarnef]

@Zeegaan we are also seeing this error sometimes when deploying to Development environment where deployment extraction fails and clear signatures + new extraction with Deploy dashboard isn't sufficient.

A recycle of the environment seems to fix it. I guess it is a similar issue in cache refreshing when Deploy tries to save the conntent type.

[anthonycodedynamo]

I'm seeing this on a local dev environment with a local SQL Server when running a tight loop to unpublish 1000s of nodes. I suspect it has something to do with GetRemainingTimeoutSeconds and the logged in user's remaining session time.

I found after rebooting IIS and keeping the session a live I managed to side-step the error. I can't find documentation on Umbraco 10+ on how to extend the session time.

Also, I'd be interested on more details of DistributedLockingReadLockDefaultTimeout and DistributedLockingWriteLockDefaultTimeout, presuming these are database row lock timeouts.

Not trying to hijack the issue here... please ignore if this isn't relevant to the issue at hand.

[russellshome]

It seems that this query is closely associated with this issue and at times takes an inordinately long time to complete

exec sp_executesql N'UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0',N'@0 int',@0=-333

[russellshome]

I am able to get the problem to happen every time if I publish the home page and then publish another page. I ran SQL Profiler and these queries while umbraco was trying to save the other page:

select cmd,* from sys.sysprocesses
where blocked > 0

SELECT * FROM sys.dm_tran_locks
  WHERE resource_database_id = DB_ID()
  AND resource_associated_entity_id = OBJECT_ID(N'dbo.umbracoLock');

That revealed a query locking the umbracoLock table that took over 25 seconds to complete - see below.

I'm not sure yet what is running this query or why but it seems like it tries to get every property of every page. In my case over 15000 rows

exec sp_executesql N'SELECT [umbracoPropertyData].[id] AS [Id], [umbracoPropertyData].[versionId] AS [VersionId], [umbracoPropertyData].[propertyTypeId] AS [PropertyTypeId], [umbracoPropertyData].[languageId] AS [LanguageId], [umbracoPropertyData].[segment] AS [Segment], [umbracoPropertyData].[intValue] AS [IntegerValue], [umbracoPropertyData].[decimalValue] AS [DecimalValue], [umbracoPropertyData].[dateValue] AS [DateValue], [umbracoPropertyData].[varcharValue] AS [VarcharValue], [umbracoPropertyData].[textValue] AS [TextValue]
FROM [umbracoPropertyData]
WHERE ([umbracoPropertyData].[versionId] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8,@9,@10,@11,@12,@13,@14,@15,@16,@17,@18,@19,@20,@21,@22,@23,@24,@25,@26,@27,@28,@29,@30,@31,@32,@33,@34,@35,@36,@37,@38,@39,@40,@41,@42,@43,@44,@45,@46,@47,@48,@49,@50,@51,@52,@53,@54,@55,@56,@57,@58,@59,@60,@61,@62,@63,@64,@65,@66,@67,@68,@69,@70,@71,@72,@73,@74,@75,@76,@77,@78,@79,@80,@81,@82,@83,@84,@85,@86,@87,@88,@89,@90,@91,@92,@93,@94,@95,@96,@97,@98,@99,@100,@101,@102,@103,@104,@105,@106,@107,@108,@109,@110,@111,@112,@113,@114,@115,@116,@117,@118,@119,@120,@121,@122,@123,@124,@125,@126,@127,@128,@129,@130,@131,@132,@133,@134,@135,@136,@137,@138,@139,@140,@141,@142,@143,@144,@145,@146,@147,@148,@149,@150,@151,@152,@153,@154,@155,@156,@157,@158,@159,@160,@161,@162,@163,@164,@165,@166,@167,@168,@169,@170,@171,@172,@173,@174,@175,@176,@177,@178,@179,@180,@181,@182,@183,@184,@185,@186,@187,@188,@189,@190,@191,@192,@193,@194,@195,@196,@197,@198,@199,@200,@201,@202,@203,@204,@205,@206,@207,@208,@209,@210,@211,@212,@213,@214,@215,@216,@217,@218,@219,@220,@221,@222,@223,@224,@225,@226,@227,@228,@229,@230,@231,@232,@233,@234,@235,@236,@237,@238,@239,@240,@241,@242,@243,@244,@245,@246,@247,@248,@249,@250,@251,@252,@253,@254,@255,@256,@257,@258,@259,@260,@261,@262,@263,@264,@265,@266,@267,@268,@269,@270,@271,@272,@273,@274,@275,@276,@277,@278,@279,@280,@281,@282,@283,@284,@285,@286,@287,@288,@289,@290,@291,@292,@293,@294,@295,@296,@297,@298,@299,@300,@301,@302,@303,@304,@305,@306,@307,@308,@309,@310,@311,@312,@313,@314,@315,@316,@317,@318,@319,@320,@321,@322,@323,@324,@325,@326,@327,@328,@329,@330,@331,@332,@333,@334,@335,@336,@337,@338,@339,@340,@341,@342,@343,@344,@345,@346,@347,@348,@349,@350,@351,@352,@353,@354,@355,@356,@357,@358,@359,@360,@361,@362,@363,@364,@365,@366,@367,@368,@369,@370,@371,@372,@373,@374,@375,@376,@377,@378,@379,@380,@381,@382,@383,@384,@385,@386,@387,@388,@389,@390,@391,@392,@393,@394,@395,@396,@397,@398,@399,@400,@401,@402,@403,@404,@405,@406,@407,@408,@409,@410,@411,@412,@413,@414,@415,@416,@417,@418,@419,@420,@421,@422,@423,@424,@425,@426,@427,@428,@429,@430,@431,@432,@433,@434,@435,@436,@437,@438,@439,@440,@441,@442,@443,@444,@445,@446,@447,@448,@449,@450,@451,@452,@453,@454,@455,@456,@457,@458,@459,@460,@461,@462,@463,@464,@465,@466,@467,@468,@469,@470,@471,@472,@473,@474,@475,@476,@477,@478,@479,@480,@481,@482,@483,@484,@485,@486,@487,@488,@489,@490,@491,@492,@493,@494,@495,@496,@497,@498,@499,@500,@501,@502,@503,@504,@505,@506,@507,@508,@509,@510,@511,@512,@513,@514,@515,@516,@517,@518,@519,@520,@521,@522,@523,@524,@525,@526,@527,@528,@529,@530,@531,@532,@533,@534,@535,@536,@537,@538,@539,@540,@541,@542,@543,@544,@545,@546,@547,@548,@549,@550,@551,@552,@553,@554,@555,@556,@557,@558,@559,@560,@561,@562,@563,@564,@565,@566,@567,@568,@569,@570,@571,@572,@573,@574,@575,@576,@577,@578,@579,@580,@581,@582,@583,@584,@585,@586,@587,@588,@589,@590,@591,@592,@593,@594,@595,@596,@597,@598,@599,@600,@601,@602,@603,@604,@605,@606,@607,@608,@609,@610,@611,@612,@613,@614,@615,@616,@617,@618,@619,@620,@621,@622,@623,@624,@625,@626,@627,@628,@629,@630,@631,@632,@633,@634,@635,@636,@637,@638,@639,@640,@641,@642,@643,@644,@645,@646,@647,@648,@649,@650,@651,@652,@653,@654,@655,@656,@657,@658,@659,@660,@661,@662,@663,@664,@665,@666,@667,@668,@669,@670,@671,@672,@673,@674,@675,@676,@677,@678,@679,@680,@681,@682,@683,@684,@685,@686,@687,@688,@689,@690,@691,@692,@693,@694,@695,@696,@697,@698,@699,@700,@701,@702,@703,@704,@705,@706,@707,@708,@709,@710,@711,@712,@713,@714,@715,@716,@717,@718,@719,@720,@721,@722,@723,@724,@725,@726,@727,@728,@729,@730,@731,@732,@733,@734,@735,@736,@737,@738,@739,@740,@741,@742,@743,@744,@745,@746,@747,@748,@749,@750,@751,@752,@753,@754,@755,@756,@757,@758,@759,@760,@761,@762,@763,@764,@765,@766,@767,@768,@769,@770,@771,@772,@773,@774,@775,@776,@777,@778,@779,@780,@781,@782,@783,@784,@785,@786,@787,@788,@789,@790,@791,@792,@793,@794,@795,@796,@797,@798,@799,@800,@801,@802,@803,@804,@805,@806,@807,@808,@809,@810,@811,@812,@813,@814,@815,@816,@817,@818,@819,@820,@821,@822,@823,@824,@825,@826,@827,@828,@829,@830,@831,@832,@833,@834,@835,@836,@837,@838,@839,@840,@841,@842,@843,@844,@845,@846,@847,@848,@849,@850,@851,@852,@853,@854,@855,@856,@857,@858,@859,@860,@861,@862,@863,@864,@865,@866,@867,@868,@869,@870,@871,@872,@873,@874,@875,@876,@877,@878,@879,@880,@881,@882,@883,@884,@885,@886,@887,@888,@889,@890,@891,@892,@893,@894,@895,@896,@897,@898,@899,@900,@901,@902,@903,@904,@905,@906,@907,@908,@909,@910,@911,@912,@913,@914,@915,@916,@917,@918,@919,@920,@921,@922,@923,@924,@925,@926,@927,@928,@929,@930,@931,@932,@933,@934,@935,@936,@937,@938,@939,@940,@941,@942,@943,@944,@945,@946,@947,@948,@949,@950,@951,@952,@953,@954,@955,@956,@957,@958,@959,@960,@961,@962,@963,@964,@965,@966,@967,@968,@969,@970,@971,@972,@973,@974,@975,@976,@977,@978,@979,@980,@981,@982,@983,@984,@985,@986,@987,@988,@989,@990,@991,@992,@993,@994,@995,@996,@997,@998,@999,@1000,@1001,@1002,@1003,@1004,@1005,@1006,@1007,@1008,@1009,@1010,@1011,@1012,@1013,@1014,@1015,@1016,@1017,@1018,@1019,@1020,@1021,@1022,@1023,@1024,@1025,@1026,@1027,@1028,@1029,@1030,@1031,@1032,@1033,@1034,@1035,@1036,@1037,@1038,@1039,@1040,@1041,@1042,@1043,@1044,@1045,@1046,@1047,@1048,@1049,@1050,@1051,@1052,@1053,@1054,@1055,@1056,@1057,@1058,@1059,@1060,@1061,@1062,@1063,@1064,@1065,@1066,@1067,@1068,@1069,@1070,@1071,@1072,@1073,@1074,@1075,@1076,@1077,@1078,@1079,@1080,@1081,@1082,@1083,@1084,@1085,@1086,@1087,@1088,@1089,@1090,@1091,@1092,@1093,@1094,@1095,@1096,@1097,@1098,@1099,@1100,@1101,@1102,@1103,@1104,@1105,@1106,@1107,@1108,@1109,@1110,@1111,@1112,@1113,@1114,@1115,@1116,@1117,@1118,@1119,@1120,@1121,@1122,@1123,@1124,@1125,@1126,@1127,@1128,@1129,@1130,@1131,@1132,@1133,@1134,@1135,@1136,@1137,@1138,@1139,@1140,@1141,@1142,@1143,@1144,@1145,@1146,@1147,@1148,@1149,@1150,@1151,@1152,@1153,@1154,@1155,@1156,@1157,@1158,@1159,@1160,@1161,@1162,@1163,@1164,@1165,@1166,@1167,@1168,@1169,@1170,@1171,@1172,@1173,@1174,@1175,@1176,@1177,@1178,@1179,@1180,@1181,@1182,@1183,@1184,@1185,@1186,@1187,@1188,@1189,@1190,@1191,@1192,@1193,@1194,@1195,@1196,@1197,@1198,@1199,@1200,@1201,@1202,@1203,@1204,@1205,@1206,@1207,@1208,@1209,@1210,@1211,@1212,@1213,@1214,@1215,@1216,@1217,@1218,@1219,@1220,@1221,@1222,@1223,@1224,@1225,@1226,@1227,@1228,@1229,@1230,@1231,@1232,@1233,@1234,@1235,@1236,@1237,@1238,@1239,@1240,@1241,@1242,@1243,@1244,@1245,@1246,@1247,@1248,@1249,@1250,@1251,@1252,@1253,@1254,@1255,@1256,@1257,@1258,@1259,@1260,@1261,@1262,@1263,@1264,@1265,@1266,@1267,@1268,@1269,@1270,@1271,@1272,@1273,@1274,@1275,@1276,@1277,@1278,@1279,@1280,@1281,@1282,@1283,@1284,@1285,@1286,@1287,@1288,@1289,@1290,@1291,@1292,@1293,@1294,@1295,@1296,@1297,@1298,@1299,@1300,@1301,@1302,@1303,@1304,@1305,@1306,@1307,@1308,@1309,@1310,@1311,@1312,@1313,@1314,@1315,@1316,@1317,@1318,@1319,@1320,@1321,@1322,@1323,@1324,@1325,@1326,@1327,@1328,@1329,@1330,@1331,@1332,@1333,@1334,@1335,@1336,@1337,@1338,@1339,@1340,@1341,@1342,@1343,@1344,@1345,@1346,@1347,@1348,@1349,@1350,@1351,@1352,@1353,@1354,@1355,@1356,@1357,@1358,@1359,@1360,@1361,@1362,@1363,@1364,@1365,@1366,@1367,@1368,@1369,@1370,@1371,@1372,@1373,@1374,@1375,@1376,@1377,@1378,@1379,@1380,@1381,@1382,@1383,@1384,@1385,@1386,@1387,@1388,@1389,@1390,@1391,@1392,@1393,@1394,@1395,@1396,@1397,@1398,@1399,@1400,@1401,@1402,@1403,@1404,@1405,@1406,@1407,@1408,@1409,@1410,@1411,@1412,@1413,@1414,@1415,@1416,@1417,@1418,@1419,@1420,@1421,@1422,@1423,@1424,@1425,@1426,@1427,@1428,@1429,@1430,@1431,@1432,@1433,@1434,@1435,@1436,@1437,@1438,@1439,@1440,@1441,@1442,@1443,@1444,@1445,@1446,@1447,@1448,@1449,@1450,@1451,@1452,@1453,@1454,@1455,@1456,@1457,@1458,@1459,@1460,@1461,@1462,@1463,@1464,@1465,@1466,@1467,@1468,@1469,@1470,@1471,@1472,@1473,@1474,@1475,@1476,@1477,@1478,@1479,@1480,@1481,@1482,@1483,@1484,@1485,@1486,@1487,@1488,@1489,@1490,@1491,@1492,@1493,@1494,@1495,@1496,@1497,@1498,@1499,@1500,@1501,@1502,@1503,@1504,@1505,@1506,@1507,@1508,@1509,@1510,@1511,@1512,@1513,@1514,@1515,@1516,@1517,@1518,@1519,@1520,@1521,@1522,@1523,@1524,@1525,@1526,@1527,@1528,@1529,@1530,@1531,@1532,@1533,@1534,@1535,@1536,@1537,@1538,@1539,@1540,@1541,@1542,@1543,@1544,@1545,@1546,@1547,@1548,@1549,@1550,@1551,@1552,@1553,@1554,@1555,@1556,@1557,@1558,@1559,@1560,@1561,@1562,@1563,@1564,@1565,@1566,@1567,@1568,@1569,@1570,@1571,@1572,@1573,@1574,@1575,@1576,@1577,@1578,@1579,@1580,@1581,@1582,@1583,@1584,@1585,@1586,@1587,@1588,@1589,@1590,@1591,@1592,@1593,@1594,@1595,@1596,@1597,@1598,@1599,@1600,@1601,@1602,@1603,@1604,@1605,@1606,@1607,@1608,@1609,@1610,@1611,@1612,@1613,@1614,@1615,@1616,@1617,@1618,@1619,@1620,@1621,@1622,@1623,@1624,@1625,@1626,@1627,@1628,@1629,@1630,@1631,@1632,@1633,@1634,@1635,@1636,@1637,@1638,@1639,@1640,@1641,@1642,@1643,@1644,@1645,@1646,@1647,@1648,@1649,@1650,@1651,@1652,@1653,@1654,@1655,@1656,@1657,@1658,@1659,@1660,@1661,@1662,@1663,@1664,@1665,@1666,@1667,@1668,@1669,@1670,@1671,@1672,@1673,@1674,@1675,@1676,@1677,@1678,@1679,@1680,@1681,@1682,@1683,@1684,@1685,@1686,@1687,@1688,@1689,@1690,@1691,@1692,@1693,@1694,@1695,@1696,@1697,@1698,@1699,@1700,@1701,@1702,@1703,@1704,@1705,@1706,@1707,@1708,@1709,@1710,@1711,@1712,@1713,@1714,@1715,@1716,@1717,@1718,@1719,@1720,@1721,@1722,@1723,@1724,@1725,@1726,@1727,@1728,@1729,@1730,@1731,@1732,@1733,@1734,@1735,@1736,@1737,@1738,@1739,@1740,@1741,@1742,@1743,@1744,@1745,@1746,@1747,@1748,@1749,@1750,@1751,@1752,@1753,@1754,@1755,@1756,@1757,@1758,@1759,@1760,@1761,@1762,@1763,@1764,@1765,@1766,@1767,@1768,@1769,@1770,@1771,@1772,@1773,@1774,@1775,@1776,@1777,@1778,@1779,@1780,@1781,@1782,@1783,@1784,@1785,@1786,@1787,@1788,@1789,@1790,@1791,@1792,@1793,@1794,@1795,@1796,@1797,@1798,@1799,@1800,@1801,@1802,@1803,@1804,@1805,@1806,@1807,@1808,@1809,@1810,@1811,@1812,@1813,@1814,@1815,@1816,@1817,@1818,@1819,@1820,@1821,@1822,@1823,@1824,@1825,@1826,@1827,@1828,@1829,@1830,@1831,@1832,@1833,@1834,@1835,@1836,@1837,@1838,@1839,@1840,@1841,@1842,@1843,@1844,@1845,@1846,@1847,@1848,@1849,@1850,@1851,@1852,@1853,@1854,@1855,@1856,@1857,@1858,@1859,@1860,@1861,@1862,@1863,@1864,@1865,@1866,@1867,@1868,@1869,@1870,@1871,@1872,@1873,@1874,@1875,@1876,@1877,@1878,@1879,@1880,@1881,@1882,@1883,@1884,@1885,@1886,@1887,@1888,@1889,@1890,@1891,@1892,@1893,@1894,@1895,@1896,@1897,@1898,@1899,@1900,@1901,@1902,@1903,@1904,@1905,@1906,@1907,@1908,@1909,@1910,@1911,@1912,@1913,@1914,@1915,@1916,@1917,@1918,@1919,@1920,@1921,@1922,@1923,@1924,@1925,@1926,@1927,@1928,@1929,@1930,@1931,@1932,@1933,@1934,@1935,@1936,@1937,@1938,@1939,@1940,@1941,@1942,@1943,@1944,@1945,@1946,@1947,@1948,@1949,@1950,@1951,@1952,@1953,@1954,@1955,@1956,@1957,@1958,@1959,@1960,@1961,@1962,@1963,@1964,@1965,@1966,@1967,@1968,@1969,@1970,@1971,@1972,@1973,@1974,@1975,@1976,@1977,@1978,@1979,@1980,@1981,@1982,@1983,@1984,@1985,@1986,@1987,@1988,@1989,@1990,@1991,@1992,@1993,@1994,@1995,@1996,@1997,@1998,@1999))',N'@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int,@9 int,@10 int,@11 int,@12 int,@13 int,@14 int,@15 int,@16 int,@17 int,@18 int,@19 int,@20 int,@21 int,@22 int,@23 int,@24 int,@25 int,@26 int,@27 int,@28 int,@29 int,@30 int,@31 int,@32 int,@33 int,@34 int,@35 int,@36 int,@37 int,@38 int,@39 int,@40 int,@41 int,@42 int,@43 int,@44 int,@45 int,@46 int,@47 int,@48 int,@49 int,@50 int,@51 int,@52 int,@53 int,@54 int,@55 int,@56 int,@57 int,@58 int,@59 int,@60 int,@61 int,@62 int,@63 int,@64 int,@65 int,@66 int,@67 int,@68 int,@69 int,@70 int,@71 int,@72 int,@73 int,@74 int,@75 int,@76 int,@77 int,@78 int,@79 int,@80 int,@81 int,@82 int,@83 int,@84 int,@85 int,@86 int,@87 int,@88 int,@89 int,@90 int,@91 int,@92 int,@93 int,@94 int,@95 int,@96 int,@97 int,@98 int,@99 int,@100 int,@101 int,@102 int,@103 int,@104 int,@105 int,@106 int,@107 int,@108 int,@109 int,@110 int,@111 int,@112 int,@113 int,@114 int,@115 int,@116 int,@117 int,@118 int,@119 int,@120 int,@121 int,@122 int,@123 int,@124 int,@125 int,@126 int,@127 int,@128 int,@129 int,@130 int,@131 int,@132 int,@133 int,@134 int,@135 int,@136 int,@137 int,@138 int,@139 int,@140 int,@141 int,@142 int,@143 int,@144 int,@145 int,@146 int,@147 int,@148 int,@149 int,@150 int,@151 int,@152 int,@153 int,@154 int,@155 int,@156 int,@157 int,@158 int,@159 int,@160 int,@161 int,@162 int,@163 int,@164 int,@165 int,@166 int,@167 int,@168 int,@169 int,@170 int,@171 int,@172 int,@173 int,@174 int,@175 int,@176 int,@177 int,@178 int,@179 int,@180 int,@181 int,@182 int,@183 int,@184 int,@185 int,@186 int,@187 int,@188 int,@189 int,@190 int,@191 int,@192 int,@193 int,@194 int,@195 int,@196 int,@197 int,@198 int,@199 int,@200 int,@201 int,@202 int,@203 int,@204 int,@205 int,@206 int,@207 int,@208 int,@209 int,@210 int,@211 int,@212 int,@213 int,@214 int,@215 int,@216 int,@217 int,@218 int,@219 int,@220 int,@221 int,@222 int,@223 int,@224 int,@225 int,@226 int,@227 int,@228 int,@229 int,@230 int,@231 int,@232 int,@233 int,@234 int,@235 int,@236 int,@237 int,@238 int,@239 int,@240 int,@241 int,@242 int,@243 int,@244 int,@245 int,@246 int,@247 int,@248 int,@249 int,@250 int,@251 int,@252 int,@253 int,@254 int,@255 int,@256 int,@257 int,@258 int,@259 int,@260 int,@261 int,@262 int,@263 int,@264 int,@265 int,@266 int,@267 int,@268 int,@269 int,@270 int,@271 int,@272 int,@273 int,@274 int,@275 int,@276 int,@277 int,@278 int,@279 int,@280 int,@281 int,@282 int,@283 int,@284 int,@285 int,@286 int,@287 int,@288 int,@289 int,@290 int,@291 int,@292 int,@293 int,@294 int,@295 int,@296 int,@297 int,@298 int,@299 int,@300 int,@301 int,@302 int,@303 int,@304 int,@305 int,@306 int,@307 int,@308 int,@309 int,@310 int,@311 int,@312 int,@313 int,@314 int,@315 int,@316 int,@317 int,@318 int,@319 int,@320 int,@321 int,@322 int,@323 int,@324 int,@325 int,@326 int,@327 int,@328 int,@329 int,@330 int,@331 int,@332 int,@333 int,@334 int,@335 int,@336 int,@337 int,@338 int,@339 int,@340 int,@341 int,@342 int,@343 int,@344 int,@345 int,@346 int,@347 int,@348 int,@349 int,@350 int,@351 int,@352 int,@353 int,@354 int,@355 int,@356 int,@357 int,@358 int,@359 int,@360 int,@361 int,@362 int,@363 int,@364 int,@365 int,@366 int,@367 int,@368 int,@369 int,@370 int,@371 int,@372 int,@373 int,@374 int,@375 int,@376 int,@377 int,@378 int,@379 int,@380 int,@381 int,@382 int,@383 int,@384 int,@385 int,@386 int,@387 int,@388 int,@389 int,@390 int,@391 int,@392 int,@393 int,@394 int,@395 int,@396 int,@397 int,@398 int,@399 int,@400 int,@401 int,@402 int,@403 int,@404 int,@405 int,@406 int,@407 int,@408 int,@409 int,@410 int,@411 int,@412 int,@413 int,@414 int,@415 int,@416 int,@417 int,@418 int,@419 int,@420 int,@421 int,@422 int,@423 int,@424 int,@425 int,@426 int,@427 int,@428 int,@429 int,@430 int,@431 int,@432 int,@433 int,@434 int,@435 int,@436 int,@437 int,@438 int,@439 int,@440 int,@441 int,@442 int,@443 int,@444 int,@445 int,@446 int,@447 int,@448 int,@449 int,@450 int,@451 int,@452 int,@453 int,@454 int,@455 int,@456 int,@457 int,@458 int,@459 int,@460 int,@461 int,@462 int,@463 int,@464 int,@465 int,@466 int,@467 int,@468 int,@469 int,@470 int,@471 int,@472 int,@473 int,@474 int,@475 int,@476 int,@477 int,@478 int,@479 int,@480 int,@481 int,@482 int,@483 int,@484 int,@485 int,@486 int,@487 int,@488 int,@489 int,@490 int,@491 int,@492 int,@493 int,@494 int,@495 int,@496 int,@497 int,@498 int,@499 int,@500 int,@501 int,@502 int,@503 int,@504 int,@505 int,@506 int,@507 int,@508 int,@509 int,@510 int,@511 int,@512 int,@513 int,@514 int,@515 int,@516 int,@517 int,@518 int,@519 int,@520 int,@521 int,@522 int,@523 int,@524 int,@525 int,@526 int,@527 int,@528 int,@529 int,@530 int,@531 int,@532 int,@533 int,@534 int,@535 int,@536 int,@537 int,@538 int,@539 int,@540 int,@541 int,@542 int,@543 int,@544 int,@545 int,@546 int,@547 int,@548 int,@549 int,@550 int,@551 int,@552 int,@553 int,@554 int,@555 int,@556 int,@557 int,@558 int,@559 int,@560 int,@561 int,@562 int,@563 int,@564 int,@565 int,@566 int,@567 int,@568 int,@569 int,@570 int,@571 int,@572 int,@573 int,@574 int,@575 int,@576 int,@577 int,@578 int,@579 int,@580 int,@581 int,@582 int,@583 int,@584 int,@585 int,@586 int,@587 int,@588 int,@589 int,@590 int,@591 int,@592 int,@593 int,@594 int,@595 int,@596 int,@597 int,@598 int,@599 int,@600 int,@601 int,@602 int,@603 int,@604 int,@605 int,@606 int,@607 int,@608 int,@609 int,@610 int,@611 int,@612 int,@613 int,@614 int,@615 int,@616 int,@617 int,@618 int,@619 int,@620 int,@621 int,@622 int,@623 int,@624 int,@625 int,@626 int,@627 int,@628 int,@629 int,@630 int,@631 int,@632 int,@633 int,@634 int,@635 int,@636 int,@637 int,@638 int,@639 int,@640 int,@641 int,@642 int,@643 int,@644 int,@645 int,@646 int,@647 int,@648 int,@649 int,@650 int,@651 int,@652 int,@653 int,@654 int,@655 int,@656 int,@657 int,@658 int,@659 int,@660 int,@661 int,@662 int,@663 int,@664 int,@665 int,@666 int,@667 int,@668 int,@669 int,@670 int,@671 int,@672 int,@673 int,@674 int,@675 int,@676 int,@677 int,@678 int,@679 int,@680 int,@681 int,@682 int,@683 int,@684 int,@685 int,@686 int,@687 int,@688 int,@689 int,@690 int,@691 int,@692 int,@693 int,@694 int,@695 int,@696 int,@697 int,@698 int,@699 int,@700 int,@701 int,@702 int,@703 int,@704 int,@705 int,@706 int,@707 int,@708 int,@709 int,@710 int,@711 int,@712 int,@713 int,@714 int,@715 int,@716 int,@717 int,@718 int,@719 int,@720 int,@721 int,@722 int,@723 int,@724 int,@725 int,@726 int,@727 int,@728 int,@729 int,@730 int,@731 int,@732 int,@733 int,@734 int,@735 int,@736 int,@737 int,@738 int,@739 int,@740 int,@741 int,@742 int,@743 int,@744 int,@745 int,@746 int,@747 int,@748 int,@749 int,@750 int,@751 int,@752 int,@753 int,@754 int,@755 int,@756 int,@757 int,@758 int,@759 int,@760 int,@761 int,@762 int,@763 int,@764 int,@765 int,@766 int,@767 int,@768 int,@769 int,@770 int,@771 int,@772 int,@773 int,@774 int,@775 int,@776 int,@777 int,@778 int,@779 int,@780 int,@781 int,@782 int,@783 int,@784 int,@785 int,@786 int,@787 int,@788 int,@789 int,@790 int,@791 int,@792 int,@793 int,@794 int,@795 int,@796 int,@797 int,@798 int,@799 int,@800 int,@801 int,@802 int,@803 int,@804 int,@805 int,@806 int,@807 int,@808 int,@809 int,@810 int,@811 int,@812 int,@813 int,@814 int,@815 int,@816 int,@817 int,@818 int,@819 int,@820 int,@821 int,@822 int,@823 int,@824 int,@825 int,@826 int,@827 int,@828 int,@829 int,@830 int,@831 int,@832 int,@833 int,@834 int,@835 int,@836 int,@837 int,@838 int,@839 int,@840 int,@841 int,@842 int,@843 int,@844 int,@845 int,@846 int,@847 int,@848 int,@849 int,@850 int,@851 int,@852 int,@853 int,@854 int,@855 int,@856 int,@857 int,@858 int,@859 int,@860 int,@861 int,@862 int,@863 int,@864 int,@865 int,@866 int,@867 int,@868 int,@869 int,@870 int,@871 int,@872 int,@873 int,@874 int,@875 int,@876 int,@877 int,@878 int,@879 int,@880 int,@881 int,@882 int,@883 int,@884 int,@885 int,@886 int,@887 int,@888 int,@889 int,@890 int,@891 int,@892 int,@893 int,@894 int,@895 int,@896 int,@897 int,@898 int,@899 int,@900 int,@901 int,@902 int,@903 int,@904 int,@905 int,@906 int,@907 int,@908 int,@909 int,@910 int,@911 int,@912 int,@913 int,@914 int,@915 int,@916 int,@917 int,@918 int,@919 int,@920 int,@921 int,@922 int,@923 int,@924 int,@925 int,@926 int,@927 int,@928 int,@929 int,@930 int,@931 int,@932 int,@933 int,@934 int,@935 int,@936 int,@937 int,@938 int,@939 int,@940 int,@941 int,@942 int,@943 int,@944 int,@945 int,@946 int,@947 int,@948 int,@949 int,@950 int,@951 int,@952 int,@953 int,@954 int,@955 int,@956 int,@957 int,@958 int,@959 int,@960 int,@961 int,@962 int,@963 int,@964 int,@965 int,@966 int,@967 int,@968 int,@969 int,@970 int,@971 int,@972 int,@973 int,@974 int,@975 int,@976 int,@977 int,@978 int,@979 int,@980 int,@981 int,@982 int,@983 int,@984 int,@985 int,@986 int,@987 int,@988 int,@989 int,@990 int,@991 int,@992 int,@993 int,@994 int,@995 int,@996 int,@997 int,@998 int,@999 int,@1000 int,@1001 int,@1002 int,@1003 int,@1004 int,@1005 int,@1006 int,@1007 int,@1008 int,@1009 int,@1010 int,@1011 int,@1012 int,@1013 int,@1014 int,@1015 int,@1016 int,@1017 int,@1018 int,@1019 int,@1020 int,@1021 int,@1022 int,@1023 int,@1024 int,@1025 int,@1026 int,@1027 int,@1028 int,@1029 int,@1030 int,@1031 int,@1032 int,@1033 int,@1034 int,@1035 int,@1036 int,@1037 int,@1038 int,@1039 int,@1040 int,@1041 int,@1042 int,@1043 int,@1044 int,@1045 int,@1046 int,@1047 int,@1048 int,@1049 int,@1050 int,@1051 int,@1052 int,@1053 int,@1054 int,@1055 int,@1056 int,@1057 int,@1058 int,@1059 int,@1060 int,@1061 int,@1062 int,@1063 int,@1064 int,@1065 int,@1066 int,@1067 int,@1068 int,@1069 int,@1070 int,@1071 int,@1072 int,@1073 int,@1074 int,@1075 int,@1076 int,@1077 int,@1078 int,@1079 int,@1080 int,@1081 int,@1082 int,@1083 int,@1084 int,@1085 int,@1086 int,@1087 int,@1088 int,@1089 int,@1090 int,@1091 int,@1092 int,@1093 int,@1094 int,@1095 int,@1096 int,@1097 int,@1098 int,@1099 int,@1100 int,@1101 int,@1102 int,@1103 int,@1104 int,@1105 int,@1106 int,@1107 int,@1108 int,@1109 int,@1110 int,@1111 int,@1112 int,@1113 int,@1114 int,@1115 int,@1116 int,@1117 int,@1118 int,@1119 int,@1120 int,@1121 int,@1122 int,@1123 int,@1124 int,@1125 int,@1126 int,@1127 int,@1128 int,@1129 int,@1130 int,@1131 int,@1132 int,@1133 int,@1134 int,@1135 int,@1136 int,@1137 int,@1138 int,@1139 int,@1140 int,@1141 int,@1142 int,@1143 int,@1144 int,@1145 int,@1146 int,@1147 int,@1148 int,@1149 int,@1150 int,@1151 int,@1152 int,@1153 int,@1154 int,@1155 int,@1156 int,@1157 int,@1158 int,@1159 int,@1160 int,@1161 int,@1162 int,@1163 int,@1164 int,@1165 int,@1166 int,@1167 int,@1168 int,@1169 int,@1170 int,@1171 int,@1172 int,@1173 int,@1174 int,@1175 int,@1176 int,@1177 int,@1178 int,@1179 int,@1180 int,@1181 int,@1182 int,@1183 int,@1184 int,@1185 int,@1186 int,@1187 int,@1188 int,@1189 int,@1190 int,@1191 int,@1192 int,@1193 int,@1194 int,@1195 int,@1196 int,@1197 int,@1198 int,@1199 int,@1200 int,@1201 int,@1202 int,@1203 int,@1204 int,@1205 int,@1206 int,@1207 int,@1208 int,@1209 int,@1210 int,@1211 int,@1212 int,@1213 int,@1214 int,@1215 int,@1216 int,@1217 int,@1218 int,@1219 int,@1220 int,@1221 int,@1222 int,@1223 int,@1224 int,@1225 int,@1226 int,@1227 int,@1228 int,@1229 int,@1230 int,@1231 int,@1232 int,@1233 int,@1234 int,@1235 int,@1236 int,@1237 int,@1238 int,@1239 int,@1240 int,@1241 int,@1242 int,@1243 int,@1244 int,@1245 int,@1246 int,@1247 int,@1248 int,@1249 int,@1250 int,@1251 int,@1252 int,@1253 int,@1254 int,@1255 int,@1256 int,@1257 int,@1258 int,@1259 int,@1260 int,@1261 int,@1262 int,@1263 int,@1264 int,@1265 int,@1266 int,@1267 int,@1268 int,@1269 int,@1270 int,@1271 int,@1272 int,@1273 int,@1274 int,@1275 int,@1276 int,@1277 int,@1278 int,@1279 int,@1280 int,@1281 int,@1282 int,@1283 int,@1284 int,@1285 int,@1286 int,@1287 int,@1288 int,@1289 int,@1290 int,@1291 int,@1292 int,@1293 int,@1294 int,@1295 int,@1296 int,@1297 int,@1298 int,@1299 int,@1300 int,@1301 int,@1302 int,@1303 int,@1304 int,@1305 int,@1306 int,@1307 int,@1308 int,@1309 int,@1310 int,@1311 int,@1312 int,@1313 int,@1314 int,@1315 int,@1316 int,@1317 int,@1318 int,@1319 int,@1320 int,@1321 int,@1322 int,@1323 int,@1324 int,@1325 int,@1326 int,@1327 int,@1328 int,@1329 int,@1330 int,@1331 int,@1332 int,@1333 int,@1334 int,@1335 int,@1336 int,@1337 int,@1338 int,@1339 int,@1340 int,@1341 int,@1342 int,@1343 int,@1344 int,@1345 int,@1346 int,@1347 int,@1348 int,@1349 int,@1350 int,@1351 int,@1352 int,@1353 int,@1354 int,@1355 int,@1356 int,@1357 int,@1358 int,@1359 int,@1360 int,@1361 int,@1362 int,@1363 int,@1364 int,@1365 int,@1366 int,@1367 int,@1368 int,@1369 int,@1370 int,@1371 int,@1372 int,@1373 int,@1374 int,@1375 int,@1376 int,@1377 int,@1378 int,@1379 int,@1380 int,@1381 int,@1382 int,@1383 int,@1384 int,@1385 int,@1386 int,@1387 int,@1388 int,@1389 int,@1390 int,@1391 int,@1392 int,@1393 int,@1394 int,@1395 int,@1396 int,@1397 int,@1398 int,@1399 int,@1400 int,@1401 int,@1402 int,@1403 int,@1404 int,@1405 int,@1406 int,@1407 int,@1408 int,@1409 int,@1410 int,@1411 int,@1412 int,@1413 int,@1414 int,@1415 int,@1416 int,@1417 int,@1418 int,@1419 int,@1420 int,@1421 int,@1422 int,@1423 int,@1424 int,@1425 int,@1426 int,@1427 int,@1428 int,@1429 int,@1430 int,@1431 int,@1432 int,@1433 int,@1434 int,@1435 int,@1436 int,@1437 int,@1438 int,@1439 int,@1440 int,@1441 int,@1442 int,@1443 int,@1444 int,@1445 int,@1446 int,@1447 int,@1448 int,@1449 int,@1450 int,@1451 int,@1452 int,@1453 int,@1454 int,@1455 int,@1456 int,@1457 int,@1458 int,@1459 int,@1460 int,@1461 int,@1462 int,@1463 int,@1464 int,@1465 int,@1466 int,@1467 int,@1468 int,@1469 int,@1470 int,@1471 int,@1472 int,@1473 int,@1474 int,@1475 int,@1476 int,@1477 int,@1478 int,@1479 int,@1480 int,@1481 int,@1482 int,@1483 int,@1484 int,@1485 int,@1486 int,@1487 int,@1488 int,@1489 int,@1490 int,@1491 int,@1492 int,@1493 int,@1494 int,@1495 int,@1496 int,@1497 int,@1498 int,@1499 int,@1500 int,@1501 int,@1502 int,@1503 int,@1504 int,@1505 int,@1506 int,@1507 int,@1508 int,@1509 int,@1510 int,@1511 int,@1512 int,@1513 int,@1514 int,@1515 int,@1516 int,@1517 int,@1518 int,@1519 int,@1520 int,@1521 int,@1522 int,@1523 int,@1524 int,@1525 int,@1526 int,@1527 int,@1528 int,@1529 int,@1530 int,@1531 int,@1532 int,@1533 int,@1534 int,@1535 int,@1536 int,@1537 int,@1538 int,@1539 int,@1540 int,@1541 int,@1542 int,@1543 int,@1544 int,@1545 int,@1546 int,@1547 int,@1548 int,@1549 int,@1550 int,@1551 int,@1552 int,@1553 int,@1554 int,@1555 int,@1556 int,@1557 int,@1558 int,@1559 int,@1560 int,@1561 int,@1562 int,@1563 int,@1564 int,@1565 int,@1566 int,@1567 int,@1568 int,@1569 int,@1570 int,@1571 int,@1572 int,@1573 int,@1574 int,@1575 int,@1576 int,@1577 int,@1578 int,@1579 int,@1580 int,@1581 int,@1582 int,@1583 int,@1584 int,@1585 int,@1586 int,@1587 int,@1588 int,@1589 int,@1590 int,@1591 int,@1592 int,@1593 int,@1594 int,@1595 int,@1596 int,@1597 int,@1598 int,@1599 int,@1600 int,@1601 int,@1602 int,@1603 int,@1604 int,@1605 int,@1606 int,@1607 int,@1608 int,@1609 int,@1610 int,@1611 int,@1612 int,@1613 int,@1614 int,@1615 int,@1616 int,@1617 int,@1618 int,@1619 int,@1620 int,@1621 int,@1622 int,@1623 int,@1624 int,@1625 int,@1626 int,@1627 int,@1628 int,@1629 int,@1630 int,@1631 int,@1632 int,@1633 int,@1634 int,@1635 int,@1636 int,@1637 int,@1638 int,@1639 int,@1640 int,@1641 int,@1642 int,@1643 int,@1644 int,@1645 int,@1646 int,@1647 int,@1648 int,@1649 int,@1650 int,@1651 int,@1652 int,@1653 int,@1654 int,@1655 int,@1656 int,@1657 int,@1658 int,@1659 int,@1660 int,@1661 int,@1662 int,@1663 int,@1664 int,@1665 int,@1666 int,@1667 int,@1668 int,@1669 int,@1670 int,@1671 int,@1672 int,@1673 int,@1674 int,@1675 int,@1676 int,@1677 int,@1678 int,@1679 int,@1680 int,@1681 int,@1682 int,@1683 int,@1684 int,@1685 int,@1686 int,@1687 int,@1688 int,@1689 int,@1690 int,@1691 int,@1692 int,@1693 int,@1694 int,@1695 int,@1696 int,@1697 int,@1698 int,@1699 int,@1700 int,@1701 int,@1702 int,@1703 int,@1704 int,@1705 int,@1706 int,@1707 int,@1708 int,@1709 int,@1710 int,@1711 int,@1712 int,@1713 int,@1714 int,@1715 int,@1716 int,@1717 int,@1718 int,@1719 int,@1720 int,@1721 int,@1722 int,@1723 int,@1724 int,@1725 int,@1726 int,@1727 int,@1728 int,@1729 int,@1730 int,@1731 int,@1732 int,@1733 int,@1734 int,@1735 int,@1736 int,@1737 int,@1738 int,@1739 int,@1740 int,@1741 int,@1742 int,@1743 int,@1744 int,@1745 int,@1746 int,@1747 int,@1748 int,@1749 int,@1750 int,@1751 int,@1752 int,@1753 int,@1754 int,@1755 int,@1756 int,@1757 int,@1758 int,@1759 int,@1760 int,@1761 int,@1762 int,@1763 int,@1764 int,@1765 int,@1766 int,@1767 int,@1768 int,@1769 int,@1770 int,@1771 int,@1772 int,@1773 int,@1774 int,@1775 int,@1776 int,@1777 int,@1778 int,@1779 int,@1780 int,@1781 int,@1782 int,@1783 int,@1784 int,@1785 int,@1786 int,@1787 int,@1788 int,@1789 int,@1790 int,@1791 int,@1792 int,@1793 int,@1794 int,@1795 int,@1796 int,@1797 int,@1798 int,@1799 int,@1800 int,@1801 int,@1802 int,@1803 int,@1804 int,@1805 int,@1806 int,@1807 int,@1808 int,@1809 int,@1810 int,@1811 int,@1812 int,@1813 int,@1814 int,@1815 int,@1816 int,@1817 int,@1818 int,@1819 int,@1820 int,@1821 int,@1822 int,@1823 int,@1824 int,@1825 int,@1826 int,@1827 int,@1828 int,@1829 int,@1830 int,@1831 int,@1832 int,@1833 int,@1834 int,@1835 int,@1836 int,@1837 int,@1838 int,@1839 int,@1840 int,@1841 int,@1842 int,@1843 int,@1844 int,@1845 int,@1846 int,@1847 int,@1848 int,@1849 int,@1850 int,@1851 int,@1852 int,@1853 int,@1854 int,@1855 int,@1856 int,@1857 int,@1858 int,@1859 int,@1860 int,@1861 int,@1862 int,@1863 int,@1864 int,@1865 int,@1866 int,@1867 int,@1868 int,@1869 int,@1870 int,@1871 int,@1872 int,@1873 int,@1874 int,@1875 int,@1876 int,@1877 int,@1878 int,@1879 int,@1880 int,@1881 int,@1882 int,@1883 int,@1884 int,@1885 int,@1886 int,@1887 int,@1888 int,@1889 int,@1890 int,@1891 int,@1892 int,@1893 int,@1894 int,@1895 int,@1896 int,@1897 int,@1898 int,@1899 int,@1900 int,@1901 int,@1902 int,@1903 int,@1904 int,@1905 int,@1906 int,@1907 int,@1908 int,@1909 int,@1910 int,@1911 int,@1912 int,@1913 int,@1914 int,@1915 int,@1916 int,@1917 int,@1918 int,@1919 int,@1920 int,@1921 int,@1922 int,@1923 int,@1924 int,@1925 int,@1926 int,@1927 int,@1928 int,@1929 int,@1930 int,@1931 int,@1932 int,@1933 int,@1934 int,@1935 int,@1936 int,@1937 int,@1938 int,@1939 int,@1940 int,@1941 int,@1942 int,@1943 int,@1944 int,@1945 int,@1946 int,@1947 int,@1948 int,@1949 int,@1950 int,@1951 int,@1952 int,@1953 int,@1954 int,@1955 int,@1956 int,@1957 int,@1958 int,@1959 int,@1960 int,@1961 int,@1962 int,@1963 int,@1964 int,@1965 int,@1966 int,@1967 int,@1968 int,@1969 int,@1970 int,@1971 int,@1972 int,@1973 int,@1974 int,@1975 int,@1976 int,@1977 int,@1978 int,@1979 int,@1980 int,@1981 int,@1982 int,@1983 int,@1984 int,@1985 int,@1986 int,@1987 int,@1988 int,@1989 int,@1990 int,@1991 int,@1992 int,@1993 int,@1994 int,@1995 int,@1996 int,@1997 int,@1998 int,@1999 int',@0=95498,@1=95499,@2=95500,@3=95501,@4=95502,@5=95503,@6=95504,@7=95505,@8=95506,@9=95507,@10=95508,@11=120653,@12=120629,@13=118045,@14=115830,@15=103690,@16=95512,@17=119947,@18=119188,@19=107235,@20=104835,@21=119948,@22=119189,@23=118048,@24=103692,@25=104836,@26=95522,@27=118049,@28=95524,@29=95526,@30=95525,@31=95527,@32=115817,@33=115816,@34=118054,@35=115838,@36=118053,@37=115837,@38=119804,@39=118050,@40=118712,@41=118051,@42=116709,@43=115884,@44=103698,@45=95539,@46=111220,@47=95541,@48=95543,@49=95542,@50=112465,@51=112464,@52=119110,@53=112055,@54=95548,@55=115845,@56=115035,@57=118301,@58=118052,@59=119949,@60=119263,@61=119734,@62=119733,@63=108435,@64=95554,@65=104837,@66=95556,@67=95558,@68=95557,@69=102649,@70=95560,@71=95561,@72=95562,@73=95564,@74=95563,@75=112162,@76=95566,@77=104838,@78=95568,@79=119109,@80=103701,@81=113544,@82=104839,@83=95574,@84=113278,@85=114398,@86=114301,@87=120507,@88=120440,@89=113831,@90=113662,@91=104840,@92=113589,@93=113547,@94=115045,@95=113718,@96=114461,@97=114093,@98=106289,@99=115331,@100=115330,@101=113722,@102=113721,@103=95593,@104=120298,@105=119201,@106=115589,@107=113553,@108=113551,@109=104842,@110=113689,@111=113586,@112=120449,@113=120448,@114=120455,@115=120453,@116=120457,@117=120456,@118=106644,@119=106184,@120=120469,@121=120460,@122=118321,@123=118320,@124=114946,@125=114945,@126=95611,@127=103711,@128=103712,@129=113584,@130=113583,@131=120317,@132=120314,@133=95620,@134=95621,@135=113431,@136=104843,@137=95624,@138=113706,@139=113615,@140=113704,@141=113576,@142=117216,@143=113702,@144=119053,@145=119051,@146=120309,@147=113668,@148=113457,@149=113456,@150=120508,@151=120240,@152=113566,@153=112659,@154=103718,@155=113591,@156=96236,@157=119143,@158=118173,@159=95646,@160=95645,@161=95648,@162=95647,@163=95650,@164=95649,@165=95652,@166=95651,@167=95654,@168=95653,@169=107916,@170=107915,@171=107919,@172=107918,@173=107922,@174=107921,@175=109038,@176=109037,@177=108578,@178=108577,@179=110413,@180=110412,@181=109403,@182=109402,@183=109674,@184=109401,@185=109675,@186=109405,@187=110627,@188=110626,@189=110407,@190=110406,@191=110405,@192=110404,@193=111860,@194=111859,@195=111858,@196=111857,@197=112491,@198=112490,@199=112489,@200=112488,@201=95656,@202=95655,@203=95658,@204=95657,@205=95660,@206=95659,@207=95662,@208=95661,@209=95664,@210=95663,@211=95666,@212=95665,@213=95668,@214=95667,@215=95670,@216=95669,@217=95672,@218=95671,@219=95674,@220=95673,@221=95676,@222=95675,@223=95678,@224=95677,@225=95680,@226=95679,@227=95682,@228=95681,@229=95684,@230=95683,@231=95686,@232=95685,@233=95688,@234=95687,@235=95690,@236=95689,@237=95692,@238=95691,@239=95694,@240=95693,@241=95696,@242=95695,@243=95698,@244=95697,@245=95700,@246=95699,@247=95702,@248=95701,@249=95704,@250=95703,@251=95706,@252=95705,@253=95708,@254=95707,@255=95710,@256=95709,@257=103720,@258=95712,@259=103721,@260=95714,@261=103722,@262=95716,@263=95718,@264=95717,@265=95720,@266=95719,@267=95722,@268=95721,@269=95724,@270=95723,@271=95726,@272=95725,@273=95728,@274=95727,@275=95730,@276=95729,@277=95732,@278=95731,@279=95734,@280=95733,@281=95736,@282=95735,@283=95738,@284=95737,@285=95740,@286=95739,@287=95742,@288=95741,@289=103723,@290=95746,@291=95748,@292=95747,@293=95750,@294=95749,@295=95752,@296=95751,@297=95754,@298=95753,@299=95756,@300=95755,@301=95758,@302=95757,@303=95760,@304=95759,@305=95762,@306=95761,@307=95764,@308=95763,@309=95766,@310=95765,@311=95768,@312=95767,@313=95770,@314=95769,@315=95772,@316=95771,@317=95774,@318=95773,@319=95776,@320=95775,@321=95778,@322=95777,@323=95780,@324=95779,@325=95782,@326=95781,@327=95784,@328=95783,@329=95786,@330=95785,@331=95788,@332=95787,@333=95790,@334=95789,@335=95792,@336=95791,@337=95794,@338=95793,@339=95796,@340=95795,@341=95798,@342=95797,@343=95800,@344=95799,@345=95802,@346=95801,@347=95804,@348=95803,@349=95806,@350=95805,@351=95808,@352=95807,@353=95810,@354=95809,@355=95812,@356=95811,@357=95814,@358=95813,@359=95816,@360=95815,@361=95818,@362=95817,@363=95820,@364=95819,@365=107909,@366=107908,@367=107907,@368=107906,@369=95822,@370=95821,@371=95824,@372=95823,@373=95826,@374=95825,@375=95828,@376=95827,@377=95830,@378=95829,@379=95832,@380=95831,@381=95834,@382=95833,@383=95836,@384=95835,@385=95838,@386=95837,@387=95840,@388=95839,@389=95842,@390=95841,@391=95844,@392=95843,@393=95846,@394=95845,@395=95848,@396=95847,@397=95850,@398=95849,@399=95852,@400=95851,@401=95854,@402=95853,@403=95856,@404=95855,@405=95858,@406=95857,@407=95860,@408=95859,@409=95862,@410=95861,@411=95864,@412=95863,@413=95866,@414=95865,@415=95868,@416=95867,@417=95870,@418=95869,@419=95872,@420=95871,@421=95874,@422=95873,@423=95876,@424=95875,@425=95878,@426=95877,@427=95880,@428=95879,@429=95884,@430=95883,@431=95886,@432=95885,@433=95888,@434=95887,@435=95890,@436=95889,@437=95892,@438=95891,@439=95894,@440=95893,@441=95896,@442=95895,@443=95898,@444=95897,@445=95900,@446=95899,@447=95902,@448=95901,@449=95904,@450=95903,@451=95906,@452=95905,@453=95908,@454=95907,@455=95910,@456=95909,@457=95912,@458=95911,@459=95914,@460=95913,@461=95916,@462=95915,@463=95918,@464=95917,@465=95920,@466=95919,@467=95922,@468=95921,@469=95924,@470=95923,@471=95926,@472=95925,@473=95928,@474=95927,@475=95930,@476=95929,@477=95932,@478=95931,@479=95934,@480=95933,@481=95936,@482=95935,@483=95938,@484=95937,@485=95940,@486=95939,@487=95942,@488=95941,@489=95944,@490=95943,@491=95946,@492=95945,@493=95948,@494=95947,@495=95950,@496=95949,@497=95952,@498=95951,@499=95954,@500=95953,@501=95956,@502=95955,@503=95958,@504=95957,@505=95960,@506=95959,@507=95962,@508=95961,@509=95964,@510=95963,@511=95966,@512=95965,@513=95968,@514=95967,@515=95970,@516=95969,@517=95972,@518=95971,@519=95974,@520=95973,@521=95976,@522=95975,@523=95978,@524=95977,@525=95980,@526=95979,@527=95982,@528=95981,@529=95984,@530=95983,@531=95986,@532=95985,@533=95988,@534=95987,@535=95990,@536=95989,@537=95992,@538=95991,@539=95994,@540=95993,@541=95996,@542=95995,@543=95998,@544=95997,@545=96000,@546=95999,@547=96002,@548=96001,@549=96004,@550=96003,@551=96006,@552=96005,@553=96008,@554=96007,@555=96010,@556=96009,@557=96012,@558=96011,@559=96014,@560=96013,@561=96016,@562=96015,@563=96018,@564=96017,@565=96020,@566=96019,@567=96022,@568=96021,@569=96024,@570=96023,@571=96026,@572=96025,@573=96028,@574=96027,@575=96030,@576=96029,@577=96032,@578=96031,@579=96034,@580=96033,@581=96036,@582=96035,@583=96038,@584=96037,@585=96040,@586=96039,@587=96042,@588=96041,@589=96044,@590=96043,@591=96046,@592=96045,@593=96048,@594=96047,@595=96050,@596=96049,@597=96052,@598=96051,@599=96054,@600=96053,@601=96056,@602=96055,@603=96058,@604=96057,@605=96060,@606=96059,@607=96062,@608=96061,@609=96064,@610=96063,@611=96066,@612=96065,@613=96068,@614=96067,@615=96070,@616=96069,@617=96072,@618=96071,@619=96074,@620=96073,@621=96076,@622=96075,@623=96078,@624=96077,@625=96080,@626=96079,@627=96082,@628=96081,@629=96084,@630=96083,@631=96086,@632=96085,@633=96088,@634=96087,@635=96090,@636=96089,@637=96092,@638=96091,@639=96094,@640=96093,@641=96096,@642=96095,@643=96098,@644=96097,@645=96100,@646=96099,@647=96102,@648=96101,@649=96104,@650=96103,@651=96106,@652=96105,@653=96108,@654=96107,@655=96110,@656=96109,@657=96112,@658=96111,@659=96114,@660=96113,@661=96116,@662=96115,@663=96118,@664=96117,@665=96120,@666=96119,@667=96122,@668=96121,@669=96124,@670=96123,@671=96126,@672=96125,@673=96128,@674=96127,@675=96130,@676=96129,@677=96132,@678=96131,@679=96134,@680=96133,@681=96136,@682=96135,@683=96138,@684=96137,@685=96140,@686=96139,@687=96142,@688=96141,@689=96144,@690=96143,@691=96146,@692=96145,@693=96148,@694=96147,@695=96150,@696=96149,@697=96152,@698=96151,@699=96154,@700=96153,@701=96156,@702=96155,@703=96158,@704=96157,@705=96160,@706=96159,@707=96162,@708=96161,@709=96164,@710=96163,@711=96166,@712=96165,@713=96168,@714=96167,@715=96170,@716=96169,@717=96172,@718=96171,@719=96174,@720=96173,@721=96176,@722=96175,@723=96178,@724=96177,@725=96180,@726=96179,@727=96182,@728=96181,@729=96184,@730=96183,@731=96186,@732=96185,@733=96188,@734=96187,@735=103724,@736=96190,@737=96192,@738=96191,@739=96194,@740=96193,@741=96196,@742=96195,@743=96198,@744=96197,@745=96200,@746=96199,@747=96202,@748=96201,@749=96204,@750=96203,@751=96206,@752=96205,@753=96208,@754=96207,@755=96210,@756=96209,@757=96212,@758=96211,@759=96214,@760=96213,@761=96216,@762=96215,@763=96218,@764=96217,@765=96220,@766=96219,@767=96222,@768=96221,@769=96224,@770=96223,@771=96226,@772=96225,@773=96228,@774=96227,@775=96230,@776=96229,@777=96232,@778=96231,@779=96234,@780=96233,@781=113172,@782=113171,@783=113174,@784=113173,@785=113175,@786=113170,@787=115020,@788=115019,@789=116480,@790=116479,@791=116478,@792=116477,@793=117699,@794=117698,@795=117924,@796=117708,@797=117925,@798=117734,@799=117712,@800=117711,@801=117920,@802=117710,@803=117921,@804=117739,@805=118153,@806=117790,@807=117922,@808=117794,@809=117944,@810=117923,@811=117716,@812=117715,@813=117914,@814=117714,@815=117915,@816=117718,@817=117916,@818=117738,@819=118154,@820=117792,@821=119140,@822=117917,@823=117918,@824=117809,@825=118155,@826=117919,@827=118735,@828=118732,@829=118738,@830=118734,@831=118737,@832=118736,@833=118740,@834=118739,@835=118742,@836=118741,@837=117722,@838=117721,@839=117724,@840=117723,@841=118753,@842=118751,@843=117895,@844=117725,@845=117896,@846=117728,@847=117897,@848=117741,@849=117945,@850=117898,@851=118156,@852=117899,@853=118157,@854=117900,@855=118158,@856=117901,@857=118159,@858=117902,@859=117903,@860=117838,@861=117904,@862=117840,@863=119141,@864=117905,@865=117906,@866=117844,@867=117907,@868=117847,@869=118161,@870=117908,@871=117909,@872=117851,@873=118162,@874=117910,@875=118163,@876=117911,@877=117912,@878=117859,@879=117913,@880=117872,@881=118744,@882=118743,@883=118749,@884=118746,@885=118750,@886=118748,@887=118754,@888=118752,@889=118756,@890=118755,@891=118758,@892=118757,@893=119478,@894=119477,@895=119738,@896=119735,@897=117732,@898=117731,@899=117892,@900=117730,@901=117893,@902=117743,@903=117894,@904=117804,@905=117929,@906=117928,@907=117802,@908=117801,@909=117946,@910=117890,@911=117891,@912=117818,@913=117881,@914=117880,@915=118169,@916=117927,@917=120668,@918=120667,@919=120511,@920=120510,@921=117813,@922=117812,@923=117888,@924=117878,@925=118168,@926=117889,@927=117882,@928=117814,@929=117883,@930=117816,@931=118160,@932=117884,@933=118166,@934=117885,@935=118167,@936=117886,@937=117887,@938=117877,@939=118165,@940=118164,@941=117933,@942=117932,@943=118170,@944=117931,@945=117937,@946=117936,@947=118171,@948=117935,@949=117260,@950=117259,@951=118469,@952=113730,@953=112758,@954=107137,@955=113608,@956=112759,@957=96245,@958=115050,@959=120439,@960=114646,@961=113557,@962=113459,@963=120311,@964=120310,@965=119568,@966=117186,@967=113560,@968=113465,@969=113724,@970=113617,@971=113727,@972=113726,@973=115051,@974=115049,@975=120060,@976=96249,@977=96251,@978=96250,@979=96253,@980=96252,@981=103728,@982=103167,@983=112610,@984=103729,@985=106212,@986=106201,@987=106441,@988=106247,@989=107310,@990=107232,@991=96259,@992=96258,@993=102660,@994=96261,@995=103170,@996=103169,@997=103731,@998=103730,@999=103732,@1000=102663,@1001=107390,@1002=103733,@1003=103734,@1004=102665,@1005=96273,@1006=96272,@1007=103735,@1008=103179,@1009=118082,@1010=103736,@1011=117659,@1012=109662,@1013=103185,@1014=103184,@1015=103738,@1016=102669,@1017=102670,@1018=96285,@1019=103739,@1020=102671,@1021=112608,@1022=103740,@1023=96291,@1024=96290,@1025=103741,@1026=102673,@1027=102674,@1028=96295,@1029=103742,@1030=103191,@1031=103743,@1032=102675,@1033=103744,@1034=102676,@1035=96303,@1036=96302,@1037=102677,@1038=96305,@1039=103194,@1040=103193,@1041=103745,@1042=103197,@1043=103746,@1044=103200,@1045=103747,@1046=102681,@1047=96315,@1048=96314,@1049=115037,@1050=102682,@1051=103748,@1052=103203,@1053=103749,@1054=103206,@1055=102685,@1056=96323,@1057=115036,@1058=102686,@1059=96326,@1060=96328,@1061=96327,@1062=115042,@1063=102687,@1064=115041,@1065=103750,@1066=96334,@1067=96333,@1068=103212,@1069=103211,@1070=115040,@1071=103751,@1072=103752,@1073=102691,@1074=103753,@1075=96342,@1076=103215,@1077=103214,@1078=115039,@1079=103754,@1080=115038,@1081=103755,@1082=112137,@1083=103756,@1084=96352,@1085=96351,@1086=103757,@1087=103221,@1088=102697,@1089=96356,@1090=117660,@1091=109659,@1092=102699,@1093=96360,@1094=102700,@1095=96362,@1096=106970,@1097=106969,@1098=106978,@1099=106968,@1100=107012,@1101=107011,@1102=107296,@1103=107295,@1104=107386,@1105=107385,@1106=107388,@1107=107384,@1108=107598,@1109=107597,@1110=107625,@1111=107604,@1112=107727,@1113=107682,@1114=107815,@1115=107814,@1116=107837,@1117=107830,@1118=107848,@1119=107846,@1120=107894,@1121=107853,@1122=114255,@1123=107949,@1124=107967,@1125=107966,@1126=108125,@1127=108104,@1128=112005,@1129=111464,@1130=108383,@1131=108379,@1132=109655,@1133=108433,@1134=108500,@1135=108499,@1136=108646,@1137=108514,@1138=108588,@1139=108559,@1140=108570,@1141=108568,@1142=112004,@1143=108681,@1144=108845,@1145=108806,@1146=109014,@1147=109013,@1148=96364,@1149=96363,@1150=96366,@1151=96365,@1152=102701,@1153=96368,@1154=102702,@1155=96370,@1156=103759,@1157=96372,@1158=103760,@1159=103227,@1160=96376,@1161=96375,@1162=117648,@1163=109654,@1164=103762,@1165=102704,@1166=107487,@1167=103763,@1168=106290,@1169=103764,@1170=96386,@1171=96385,@1172=103765,@1173=102707,@1174=103766,@1175=103233,@1176=103236,@1177=103235,@1178=103767,@1179=103239,@1180=96396,@1181=96395,@1182=103242,@1183=103241,@1184=96400,@1185=96399,@1186=103768,@1187=103245,@1188=103769,@1189=96404,@1190=102712,@1191=96406,@1192=103770,@1193=102713,@1194=103771,@1195=96410,@1196=103251,@1197=103250,@1198=103772,@1199=102714,@1200=102715,@1201=96416,@1202=103254,@1203=103253,@1204=96420,@1205=96419,@1206=103773,@1207=102717,@1208=117037,@1209=103774,@1210=102719,@1211=96426,@1212=96428,@1213=96427,@1214=103775,@1215=102720,@1216=103257,@1217=103256,@1218=103776,@1219=102721,@1220=96436,@1221=96435,@1222=103777,@1223=102722,@1224=103778,@1225=103260,@1226=103779,@1227=102724,@1228=103780,@1229=102725,@1230=103263,@1231=103262,@1232=96448,@1233=96447,@1234=103266,@1235=103265,@1236=103781,@1237=102727,@1238=103782,@1239=96454,@1240=103783,@1241=102728,@1242=103784,@1243=102729,@1244=96460,@1245=96459,@1246=103785,@1247=102730,@1248=117662,@1249=117661,@1250=103269,@1251=103268,@1252=103787,@1253=103272,@1254=96470,@1255=96469,@1256=103788,@1257=103275,@1258=102735,@1259=96474,@1260=103789,@1261=96476,@1262=103790,@1263=102736,@1264=103791,@1265=102737,@1266=102738,@1267=96482,@1268=102739,@1269=96484,@1270=103792,@1271=102740,@1272=96488,@1273=96487,@1274=103793,@1275=102741,@1276=103794,@1277=102742,@1278=103795,@1279=102743,@1280=102744,@1281=96496,@1282=103796,@1283=102745,@1284=103799,@1285=103798,@1286=103800,@1287=102747,@1288=96504,@1289=96503,@1290=103801,@1291=102748,@1292=103802,@1293=102749,@1294=103803,@1295=103278,@1296=103804,@1297=102750,@1298=112611,@1299=106363,@1300=96516,@1301=96515,@1302=103806,@1303=96518,@1304=102752,@1305=96520,@1306=96522,@1307=96521,@1308=96524,@1309=96523,@1310=103807,@1311=102753,@1312=103808,@1313=102754,@1314=102755,@1315=96530,@1316=102756,@1317=96532,@1318=103809,@1319=96534,@1320=103810,@1321=103281,@1322=103284,@1323=103283,@1324=103287,@1325=103286,@1326=96542,@1327=96541,@1328=103811,@1329=103290,@1330=106348,@1331=103812,@1332=103813,@1333=103293,@1334=103814,@1335=102762,@1336=103815,@1337=102763,@1338=96554,@1339=96553,@1340=103816,@1341=102764,@1342=103817,@1343=102765,@1344=103296,@1345=103295,@1346=103818,@1347=102767,@1348=96564,@1349=96563,@1350=102768,@1351=96566,@1352=112138,@1353=112074,@1354=103299,@1355=103298,@1356=96572,@1357=96571,@1358=103819,@1359=102770,@1360=103820,@1361=102771,@1362=102772,@1363=96578,@1364=103302,@1365=103301,@1366=102774,@1367=96582,@1368=103821,@1369=103305,@1370=103822,@1371=102776,@1372=96588,@1373=96587,@1374=103823,@1375=102777,@1376=102778,@1377=96592,@1378=102779,@1379=96594,@1380=102780,@1381=96596,@1382=102781,@1383=96598,@1384=103824,@1385=102782,@1386=102783,@1387=96602,@1388=102784,@1389=96604,@1390=103825,@1391=102785,@1392=102786,@1393=96608,@1394=96610,@1395=96609,@1396=103826,@1397=102787,@1398=102788,@1399=96614,@1400=102789,@1401=96616,@1402=103827,@1403=102790,@1404=111376,@1405=103828,@1406=102792,@1407=96622,@1408=103829,@1409=102793,@1410=102794,@1411=96626,@1412=102795,@1413=96628,@1414=102796,@1415=96630,@1416=96632,@1417=96631,@1418=102797,@1419=96634,@1420=103830,@1421=102798,@1422=102799,@1423=96638,@1424=103831,@1425=102800,@1426=103832,@1427=102801,@1428=103833,@1429=102802,@1430=102803,@1431=96646,@1432=96648,@1433=96647,@1434=110230,@1435=103834,@1436=103835,@1437=102805,@1438=102806,@1439=96654,@1440=102807,@1441=96656,@1442=102808,@1443=96658,@1444=102809,@1445=96660,@1446=103836,@1447=102810,@1448=96664,@1449=96663,@1450=102811,@1451=96666,@1452=102812,@1453=96668,@1454=103837,@1455=96670,@1456=103838,@1457=96672,@1458=102813,@1459=96674,@1460=103839,@1461=102814,@1462=96678,@1463=96677,@1464=103840,@1465=102815,@1466=102816,@1467=96682,@1468=102817,@1469=96684,@1470=103841,@1471=102818,@1472=102819,@1473=96688,@1474=102820,@1475=96690,@1476=102821,@1477=96692,@1478=102822,@1479=96694,@1480=96696,@1481=96695,@1482=103842,@1483=102823,@1484=102824,@1485=96700,@1486=102825,@1487=96702,@1488=102826,@1489=96704,@1490=103843,@1491=102827,@1492=102828,@1493=96708,@1494=96710,@1495=96709,@1496=96712,@1497=96711,@1498=103844,@1499=102829,@1500=103845,@1501=96716,@1502=102830,@1503=96718,@1504=102831,@1505=96720,@1506=102832,@1507=96722,@1508=102833,@1509=96724,@1510=102834,@1511=96726,@1512=103846,@1513=102835,@1514=102836,@1515=96730,@1516=102837,@1517=96732,@1518=96734,@1519=96733,@1520=102838,@1521=96736,@1522=103847,@1523=102839,@1524=103848,@1525=102840,@1526=102841,@1527=96742,@1528=96744,@1529=96743,@1530=102842,@1531=96746,@1532=102843,@1533=96748,@1534=103849,@1535=96750,@1536=103850,@1537=102844,@1538=96754,@1539=96753,@1540=102845,@1541=96756,@1542=102846,@1543=96758,@1544=103851,@1545=102847,@1546=102848,@1547=96762,@1548=102849,@1549=96764,@1550=102850,@1551=96766,@1552=103852,@1553=102851,@1554=103853,@1555=102852,@1556=96772,@1557=96771,@1558=102853,@1559=96774,@1560=96776,@1561=96775,@1562=102854,@1563=96778,@1564=96780,@1565=96779,@1566=102855,@1567=96782,@1568=103854,@1569=102856,@1570=102857,@1571=96786,@1572=102858,@1573=96788,@1574=103855,@1575=102859,@1576=102860,@1577=96792,@1578=96794,@1579=96793,@1580=102861,@1581=96796,@1582=102862,@1583=96798,@1584=96800,@1585=96799,@1586=102863,@1587=96802,@1588=102864,@1589=96804,@1590=106334,@1591=103856,@1592=103857,@1593=102866,@1594=102867,@1595=96810,@1596=103858,@1597=102868,@1598=103859,@1599=102869,@1600=103860,@1601=102870,@1602=96818,@1603=96817,@1604=102871,@1605=96820,@1606=103861,@1607=102872,@1608=102873,@1609=96824,@1610=102874,@1611=96826,@1612=103862,@1613=102875,@1614=102876,@1615=96830,@1616=102877,@1617=96832,@1618=96834,@1619=96833,@1620=103863,@1621=102878,@1622=102879,@1623=96838,@1624=102880,@1625=96840,@1626=108552,@1627=103864,@1628=103865,@1629=102882,@1630=102883,@1631=96846,@1632=102884,@1633=96848,@1634=103866,@1635=102885,@1636=102886,@1637=96852,@1638=102887,@1639=96854,@1640=96856,@1641=96855,@1642=96858,@1643=96857,@1644=103867,@1645=102888,@1646=102889,@1647=96862,@1648=103868,@1649=102890,@1650=102891,@1651=96866,@1652=102892,@1653=96868,@1654=102893,@1655=96870,@1656=102894,@1657=96872,@1658=96874,@1659=96873,@1660=103869,@1661=102895,@1662=102896,@1663=96878,@1664=102897,@1665=96880,@1666=102898,@1667=96882,@1668=102899,@1669=96884,@1670=103870,@1671=102900,@1672=96888,@1673=96887,@1674=96890,@1675=96889,@1676=103871,@1677=102901,@1678=103872,@1679=102902,@1680=96896,@1681=96895,@1682=96898,@1683=96897,@1684=102903,@1685=96900,@1686=102904,@1687=96902,@1688=116355,@1689=102905,@1690=103873,@1691=102906,@1692=102907,@1693=96908,@1694=102908,@1695=96910,@1696=103874,@1697=102909,@1698=96914,@1699=96913,@1700=102910,@1701=96916,@1702=102911,@1703=96918,@1704=102912,@1705=96920,@1706=103875,@1707=102913,@1708=102914,@1709=96924,@1710=103876,@1711=102915,@1712=102916,@1713=96928,@1714=102917,@1715=96930,@1716=103877,@1717=102918,@1718=96934,@1719=96933,@1720=102919,@1721=96936,@1722=102920,@1723=96938,@1724=102921,@1725=96940,@1726=103878,@1727=102922,@1728=96944,@1729=96943,@1730=102923,@1731=96946,@1732=102924,@1733=96948,@1734=102925,@1735=96950,@1736=102926,@1737=96952,@1738=102927,@1739=96954,@1740=96956,@1741=96955,@1742=103879,@1743=102928,@1744=103880,@1745=102929,@1746=103881,@1747=102930,@1748=103882,@1749=102931,@1750=102932,@1751=96966,@1752=103883,@1753=102933,@1754=102934,@1755=96970,@1756=102935,@1757=96972,@1758=96974,@1759=96973,@1760=103884,@1761=102936,@1762=103885,@1763=102937,@1764=103886,@1765=102938,@1766=103887,@1767=102939,@1768=103888,@1769=102940,@1770=103889,@1771=102941,@1772=103890,@1773=102942,@1774=103891,@1775=102943,@1776=102944,@1777=96992,@1778=103892,@1779=102945,@1780=96996,@1781=96995,@1782=102946,@1783=96998,@1784=102947,@1785=97000,@1786=102948,@1787=97002,@1788=102949,@1789=97004,@1790=102950,@1791=97006,@1792=103893,@1793=102951,@1794=103894,@1795=102952,@1796=102953,@1797=97012,@1798=97014,@1799=97013,@1800=102954,@1801=97016,@1802=103895,@1803=97018,@1804=102955,@1805=97020,@1806=102956,@1807=97022,@1808=97024,@1809=97023,@1810=103896,@1811=97026,@1812=102957,@1813=97028,@1814=102958,@1815=97030,@1816=102959,@1817=97032,@1818=103897,@1819=102960,@1820=102961,@1821=97036,@1822=97038,@1823=97037,@1824=102962,@1825=97040,@1826=103898,@1827=102963,@1828=107233,@1829=107110,@1830=102965,@1831=97046,@1832=102966,@1833=97048,@1834=102967,@1835=97050,@1836=102968,@1837=97052,@1838=103899,@1839=102969,@1840=103900,@1841=102970,@1842=103901,@1843=102971,@1844=103902,@1845=102972,@1846=97062,@1847=97061,@1848=102973,@1849=97064,@1850=102974,@1851=97066,@1852=102975,@1853=97068,@1854=103903,@1855=102976,@1856=102977,@1857=97072,@1858=97074,@1859=97073,@1860=103904,@1861=102978,@1862=103905,@1863=102979,@1864=102980,@1865=97080,@1866=103906,@1867=102981,@1868=102982,@1869=97084,@1870=97086,@1871=97085,@1872=97088,@1873=97087,@1874=102983,@1875=97090,@1876=103907,@1877=102984,@1878=102985,@1879=97094,@1880=102986,@1881=97096,@1882=102987,@1883=97098,@1884=97100,@1885=97099,@1886=102988,@1887=97102,@1888=97104,@1889=97103,@1890=97106,@1891=97105,@1892=97108,@1893=97107,@1894=103908,@1895=102989,@1896=102990,@1897=97112,@1898=102991,@1899=97114,@1900=102992,@1901=97116,@1902=102993,@1903=97118,@1904=97120,@1905=97119,@1906=103909,@1907=102994,@1908=102995,@1909=97124,@1910=102996,@1911=97126,@1912=102997,@1913=97128,@1914=102998,@1915=97130,@1916=102999,@1917=97132,@1918=97134,@1919=97133,@1920=103000,@1921=97136,@1922=103910,@1923=97138,@1924=103911,@1925=103001,@1926=103912,@1927=103002,@1928=103003,@1929=97144,@1930=103913,@1931=103004,@1932=103914,@1933=103005,@1934=103006,@1935=97150,@1936=103915,@1937=103007,@1938=103916,@1939=103008,@1940=97156,@1941=97155,@1942=103009,@1943=97158,@1944=103917,@1945=103010,@1946=103918,@1947=103011,@1948=103919,@1949=103012,@1950=103013,@1951=97166,@1952=97168,@1953=97167,@1954=103920,@1955=97170,@1956=103921,@1957=97172,@1958=107106,@1959=103922,@1960=97176,@1961=97175,@1962=103923,@1963=97178,@1964=103924,@1965=97180,@1966=103925,@1967=97182,@1968=103926,@1969=97184,@1970=103927,@1971=97186,@1972=103928,@1973=97188,@1974=107111,@1975=103929,@1976=103930,@1977=97192,@1978=97194,@1979=97193,@1980=103931,@1981=97196,@1982=103932,@1983=97198,@1984=103933,@1985=97200,@1986=103934,@1987=97202,@1988=103935,@1989=97204,@1990=97206,@1991=97205,@1992=103936,@1993=97208,@1994=103937,@1995=97210,@1996=103938,@1997=97212,@1998=103939,@1999=97214

[russellshome]

"This is a bugger of a query"

https://github.com/umbraco/Umbraco-CMS/blob/d9dced55c4ed86ef979b34277534fd48f852013b/src/Umbraco.Infrastructure/Persistence/Repositories/Implement/ContentRepositoryBase.cs#L755

Perhaps this issue can be solved by this query not necessitating the locking of the umbracoLock table while doing that query?

[Zeegaan]

@russellshome If you can reproduce it every time, could I get the database from you? Would help greatly in debugging the issue 👍

[russellshome]

Do you have a location where I can put it?

[Zeegaan]

You can send it to my email: nge@umbraco.dk 👌 A backpack file of the database would be great 🙌

[russellshome]

Are you sure your email will accept close to a a GB file? Most email servers will reject that sort of size. I don't think I will be able to send it let alone you receive it attached to an email.

[chriskarkowsky]

Hi,

I have the same issue running 11.4.0.

I have increased the WriteLockTimeout to 20 seconds from 5: "DistributedLockingWriteLockDefaultTimeout": "00:00:20"

After trying to Save, I always get the error:

We are setting up a load balanced environment in accordance with the documentation here: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps

We have two applications, one for backoffice and one for the frontend. We have also assigned the proper Roles for each, where frontend would be the subscriber and backoffice would be the scheduler (https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/flexible-advanced).

[chriskarkowsky]

It seems that this query is closely associated with this issue and at times takes an inordinately long time to complete

exec sp_executesql N'UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0',N'@0 int',@0=-333

I noticed this too in Long Running Queries in the Azure Dashboard. It seems like a simple toggle of the value for the ContentTree node.

Whenever the WriteLock issue appears, I tried simply running the toggle via an UPDATE command UPDATE [dbo].[umbracoLock] set [value]=-1 where [id]=-333

Should be an instantaneous result, but you can see it is taking way over the WriteLock timeout limit:

We have a 50 DTU allocated database, and we are well below max DTU usage.

[andrewmoscardino-awh]

I should chime in here. A project I work on was having this issue in production and the fix seemed to be making sure that the Lucene/Examine settings were correct for the admin and front-end app services.

The docs for running on Azure Web Apps say to set Umbraco.CMS.Examine.LuceneDirectoryFactory to SyncedTempFileSystemDirectoryFactory. What I missed on the docs for load balancing on Azure Web Apps was that only the admin site should use that value. Front-end sites should use TempFileSystemDirectoryFactory.

After that was corrected, the errors stopped.

[chriskarkowsky]

I did a bit more digging into the database, and found that my issue isn't explicitly caused by the umbracoLock table, but the way Azure may be recycling my app with CI/CD. I noticed this after recycling the app from the portal or pushing changes.

After a restart of the Umbraco app (Dockerized), I did the following.

1. Ran "exec sp_who2" in the database to get a list of clients. Noticed there was one "sleeping" for the hostname assigned by Azure for the Core Microsoft Client
2. Once the app booted, I tried saving. First time worked, second time failed for with the write lock error.
3. Running "exec sp_who2" again, I now noticed there were 2 different clients with the same hostname.
4. During clicking save again, I ran sp_who2 and noticed that the one client was SUSPENDED and being blocked by the duplicate client.

^During trying to Save in Umbraco backoffice, before a Write Lock error is thrown.

5. After this, I had to run KILL 84, and then no more write lock error!

I am not sure how to address this issue, whether Umbraco is not correctly closing the connection or with Azure not stopping the Docker image correctly. But thought I would share.

[thomashdk]

Having same issue on 11.4.2.

Have multiple solutions running on same code. But only this solution has this error.

It used around 20-40% more memory then the other solutions. This one don't have any trafic, the others have. It has a lot of nodes and media files. (94774)

I'm not running on a azure setup up. Just a windows server and a sql server

Running the Data Integrity health check fails with: Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception (10054): An existing connection was forcibly closed by the remote host.

I have these console errors also:

Is there a fix or a workaround ?

[chriskarkowsky]

Thanks for bumping this at it is still casing issues up to today. The only way I have found to workaround it is to kill the process. This isn't going to work when multiple users are trying to save in the backoffice.

I am still researching what is going on at a DB level during the save.

1. Click Save on a page in the backoffice
2. I run the following SQL cmd immediately SELECT * FROM sys.dm_exec_requests CROSS APPLY sys.dm_exec_sql_text(sql_handle)

The newly spun up process (69) is being blocked by 54

3. run dbcc inputbuffer(54) Event info shows this query: (@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int)SELECT [cmsPropertyType].[id] AS [Id], [cmsPropertyType].[dataTypeId] AS [DataTypeId], [cmsPropertyType].[contentTypeId] AS [ContentTypeId], [cmsPropertyType].[propertyTypeGroupId] AS [PropertyTypeGroupId], [cmsPropertyType].[Alias] AS [Alias], [cmsPropertyType].[Name] AS [Name], [cmsPropertyType].[sortOrder] AS [SortOrder], [cmsPropertyType].[mandatory] AS [Mandatory], [cmsPropertyType].[mandatoryMessage] AS [MandatoryMessage], [cmsPropertyType].[validationRegExp] AS [ValidationRegExp], [cmsPropertyType].[validationRegExpMessage] AS [ValidationRegExpMessage], [cmsPropertyType].[Description] AS [Description], [cmsPropertyType].[labelOnTop] AS [LabelOnTop], [cmsPropertyType].[variations] AS [Variations], [cmsPropertyType].[UniqueID] AS [UniqueId], [umbracoDataType].[nodeId] AS [DataTypeDto__NodeId], [umbracoDataType].[propertyEditorAlias] AS [DataTypeDto__EditorAlias], [umbracoDataType].[dbType] AS [DataTypeDto__DbType], [umbracoDataType].[config] AS [DataTypeDto__Configuration]FROM [cmsPropertyType]INNER JOIN [umbracoDataType]ON ([cmsPropertyType].[dataTypeId] = [umbracoDataType].[nodeId])WHERE ([cmsPropertyType].[id] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8))

This is similar to @russellshome was seeing.

4. At this point I would manually run KILL 54 to resolve the issue and allow users to continue to save.
5. I should mention that 54 is a sleeping process that needs to be killed. I don't understand how a sleeping process can be blocking the new UPDATE.

[Zeegaan]

@russellshome Sorry for leaving you unanswered for a while, I did not remember this after vacation. 😨

Hmmm if email won't do, I guess a dropbox link or something similar? 🤔

[thomashdk]

Thanks for bumping this at it is still casing issues up to today. The only way I have found to workaround it is to kill the process. This isn't going to work when multiple users are trying to save in the backoffice.

I am still researching what is going on at a DB level during the save.

1. Click Save on a page in the backoffice
2. I run the following SQL cmd immediately SELECT * FROM sys.dm_exec_requests CROSS APPLY sys.dm_exec_sql_text(sql_handle)

The newly spun up process (69) is being blocked by 54

3. run dbcc inputbuffer(54) Event info shows this query: (@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int)SELECT [cmsPropertyType].[id] AS [Id], [cmsPropertyType].[dataTypeId] AS [DataTypeId], [cmsPropertyType].[contentTypeId] AS [ContentTypeId], [cmsPropertyType].[propertyTypeGroupId] AS [PropertyTypeGroupId], [cmsPropertyType].[Alias] AS [Alias], [cmsPropertyType].[Name] AS [Name], [cmsPropertyType].[sortOrder] AS [SortOrder], [cmsPropertyType].[mandatory] AS [Mandatory], [cmsPropertyType].[mandatoryMessage] AS [MandatoryMessage], [cmsPropertyType].[validationRegExp] AS [ValidationRegExp], [cmsPropertyType].[validationRegExpMessage] AS [ValidationRegExpMessage], [cmsPropertyType].[Description] AS [Description], [cmsPropertyType].[labelOnTop] AS [LabelOnTop], [cmsPropertyType].[variations] AS [Variations], [cmsPropertyType].[UniqueID] AS [UniqueId], [umbracoDataType].[nodeId] AS [DataTypeDto__NodeId], [umbracoDataType].[propertyEditorAlias] AS [DataTypeDto__EditorAlias], [umbracoDataType].[dbType] AS [DataTypeDto__DbType], [umbracoDataType].[config] AS [DataTypeDto__Configuration]FROM [cmsPropertyType]INNER JOIN [umbracoDataType]ON ([cmsPropertyType].[dataTypeId] = [umbracoDataType].[nodeId])WHERE ([cmsPropertyType].[id] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8))

This is similar to @russellshome was seeing.

4. At this point I would manually run KILL 54 to resolve the issue and allow users to continue to save.
5. I should mention that 54 is a sleeping process that needs to be killed. I don't understand how a sleeping process can be blocking the new UPDATE.

Yeah, I'm doing the same for now...

[chriskarkowsky]

Here is a lineage from the log of 4 items when I attempted to refresh the Internal Index and getting write lock, all on the same process (69):

[chriskarkowsky]

Here are a couple more related errors:

[glenatron]

We're running into the same problem on 10.6.1 - @chriskarkowsky's workaround works to unblock the problem but it would be nice to prevent it altogether.

[SATC-Ben]

I can confirm 11.4.0 gets it too. Happens in Azure and Locally. I've seen it from Usync and our Content Import systems during dynamic saving of content.

[pvhees]

We ran into the same problem today (Umbraco 11.4.2 on Azure). In our case the media section was blocked. The workaround indeed works.

[mikefitz888]

I also had this issue occurring regularly on Umbraco 10.6.1 in Azure. I noticed it often coincided with spikes in database activity (background tasks / start-up after a deployment). Scaling the database from 50 DTU to 100 DTU has helped mitigate this in my case.

[gfyans]

My experience of this is similar to @mikefitz888's above. We have uSync installed and during startup we noticed that uSync failed to import every single time on 1 site with the "failed to acquire write lock -333" error, but ran successfully on another site with the same codebase.

I had my suspicions that it was related to the number of content nodes, so I did some testing, and these are my results:

(my test involved manually editing 1 uSync file and restarting the site to see if Umbraco would import the change)

10 tests in a site with 20k+ content nodes - all 10 failed 10 tests in the same site and DB only this time I deleted most of the content to leave approx. 100 nodes - all 10 passed

I then reset the DB to the 20k nodes and ran some tests on different tiers of database:

Standard S0: 10 DTUs - Failed Standard S1: 20 DTUs - Failed Standard S2: 50 DTUs - Failed Standard S2: 100 DTUs - Succeeded

We've scaled the database up to 100 DTUs to get rid of the error, but this is now costing 10x more than what we would normally be spending.

[chriskarkowsky]

@gfyans and @mikefitz888 Thanks for sharing this. We have attempted to scale to try to get rid of the error, but not all the way up to 100 DTU. That probably wouldn't be sustainable anyway due to the cost.

Our environment has uSync completely disabled to attempt to Import on Save and Import on Start, however yesterday I was running large imports manually, and the issue popped up again in our production environment which is at 50 DTU currently.

[misi444]

Just chiming in with my experience, it might be relevant.

I have implemented a custom endpoint which imports data from a payload - creating documents and media. I've tested it in my local environment connecting to a local SQL server running in docker. Not a single issue importing hundreds of records.

Then I tried calling the endpoint in my app which was deployed as an Azure App Service, and this error just keeps popping up. I suspected some background tasks after uploading media or something, so I implemented chunks and cooldown periods in my app which calls the endpoint with the data, but the error seems sporadic and does not seem to correlate to how much time I wait between requests.

Then I tried again running the Umbraco solution locally, but this time connecting to the database used by this Azure App Service (which is an Azure SQL Server) instead of my local docker database and the error started occurring again.

Note that my local database is a direct copy of the one in Azure SQL. I backed up and restored it just before testing the endpoint.

So this seems to be related to Azure SQL, as someone else has already suggested.

Umbraco.Cms 11.2.1

[SATC-Ben]

@misi444 We have a similar setup to you. We call in data from an API to CRUD pages in Umbraco (V11.4.0) via a nightly task. I have had the error -333 via a local SQL DB as well though.

We have not seen the issue once in a few weeks when we were seeing it quite frequently previously. I did 2 things prior to this:

• We defragged the DB indexes. Some tables were at 98% (really bad).
• Our scheduled task wasn't async but recently I refactored it so that it was. This may not have done much as the 'Save' and 'Publish' calls aren't async but it possibly helped.
• I also created a smaller scope (using ICoreScope scope = _scopeProvider.CreateCoreScope();) for the code to run. So previously we were running 2 pulls from the API in one large scope. I cut that down to one API call per scope.

If this changes I'll let you all know but hopefully, this has fixed it for us.

[chriskarkowsky]

Update from me:

I updated our production database to 100 DTU and did not see any improvement.

To reiterate our setup, we have a backend and frontend Azure WebApp, each with a staging slot. Today, we saw write lock creep up on our backend users whenever we were swapping out frontend. Then, we had to kill processes in the DB each time.

I was looking at this error specifically, which throws a -333 error to our users in the backend:

Microsoft.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)

Error Number:1222,State:51,Class:16

The way I interpret this is that Umbraco is attempting to gain a write lock ([UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0',N'@0 int',@0=-333 ?](https://github.com/umbraco/Umbraco-CMS/blob/511ee96c9e0c54480b5350fa0bcabe693bdc1e0a/src/Umbraco.Cms.Persistence.SqlServer/Services/SqlServerDistributedLockingMechanism.cs#L157)) , and failing because of the blocking process on the db level, which owns the lock on the WriteLock table itself. The killed off process, which is idle, will never relinquish the lock.

I then started researching NPoco and came across this on their wiki, which has me concerned:

Note: Database does auto close. so if you don't explicitly call OpenSharedConnection() , this usage is valid.

Note: Database needs to be disposed to close the connection (think of it as your connection object).

My theory is that Umbraco is not closing connections properly with this library. I would not know how to address this.

[bergmania]

My theory is that Umbraco is not closing connections properly with this library. I would not know how to address this.

The NPoco database is disposed when the Umbraco scope is disposed. As far as I can see, we are wrapping all scopes in using. 🤷‍♂️

[BartDM]

We're experiencing the same problems with version 10.7.0 hosted on Azure and working with Azure DB.

I did the same checks as @thomashdk (see https://github.com/umbraco/Umbraco-CMS/issues/14195#issuecomment-1691825759).

The query that was blocking seems to be (@0 int)SELECT [umbracoUserGroup2Language].[userGroupId] AS [UserGroupId], [umbracoUserGroup2Language].[languageId] AS [LanguageId] FROM [umbracoUserGroup2Language] WHERE ([umbracoUserGroup2Language].[userGroupId] IN (@0))

Killing the process released the lock and content editors were able to work.

[madsoulswe]

Received the same error when migrating a smaller site using uSync (only 151 content-nodes and 297 media items). Sadly my db user didn't have permission to check for blocking sessions so I hade to do some sql-magic to migrate.

This is a Sql Server Express 2022 running on Ubuntu 20

[chriskarkowsky]

Looks like the issue might be addressed in 12.3. What is the timeline on this release?

[bergmania]

Looks like the issue might be addressed in 12.3. What is the timeline on this release?

There will be an RC next week and release in three weeks. We are not sure this fixes the issue, but if it does, we will patch v10 too.

[r-modica-cti]

Have we been able to confirm if this resolves the issue and will be patched in 10?

[bergmania]

Have we been able to confirm if this resolves the issue and will be patched in 10?

We do not have any information yet. Ill try to put it into 10.8.0-rc so please test it out. We expect to release it tomorrow

[andersbrohall]

I've got the same issue, but on-prem. It does seem to be an issue triggered by 'simultanious' users. Duh.

[jerpenol]

We are still experiencing this issue on 12.3.1.

In our case it mostly happens on startup of the application (after a deployment), when fetching the ContentTree. The ContentTree takes a very long time to load and eventually gives a timeout exception. I suppose this is because the examine index might be being rebuilt on startup.

We also see that executing the query on the "umbracoLock" table is not immediate (we have not seen it complete) like @chriskarkowsky mentioned. We are also seeing the same blocking process.

We are running in a load balanced environment on Azure, with the following relevant config:

"Umbraco": {
  "CMS": {
    "Hosting": {
      "LocalTempStorageLocation": "EnvironmentTemp"
    },
    "Examine": {
      "LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory" //backoffice instance
      "LuceneDirectoryFactory": "TempFileSystemDirectoryFactory" //frontend instance
    },
    "Global": {
      "MainDomLock": "SqlMainDomLock"
    }
  }
}

After our client contacted the Umbraco support, we received a link to the updated Load Balancing documentation that the MainDomLock should be set to FileSystemMainDomLock, but the same timeout issues occur whilst trying to establish the read lock (SqlServerDistributedLock).

Error message

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Exception Details

Microsoft.Data.SqlClient.SqlException, Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Stacktrace

at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteScalar()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteScalarHelper(DbCommand cmd)
   at NPoco.Database.ExecuteScalar[T](String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase.ExecuteScalar[T](String sql, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainReadLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.ReadLock(Int32 lockId, Nullable`1 obtainLockTimeout)
   at Umbraco.Cms.Core.Scoping.LockingMechanism.ObtainReadLock(Int32 lockId, Nullable`1 timeout)
   at Umbraco.Cms.Core.Scoping.LockingMechanism.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)
   at Umbraco.Cms.Core.Scoping.LockingMechanism.EagerReadLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Cms.Core.Scoping.LockingMechanism.EnsureLocks(Guid scopeInstanceId)
   at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.ContentRepositoryBase`3.CountChildren(Int32 parentId, String contentTypeAlias)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.DocumentRepository.<RecycleBinSmells>b__60_0()
   at Umbraco.Extensions.AppCacheExtensions.<>c__DisplayClass5_0`1.<GetCacheItem>b__0()
   at Umbraco.Cms.Core.Cache.SafeLazy.<>c__DisplayClass1_0.<GetSafeLazy>b__0()
--- End of stack trace from previous location ---
   at Umbraco.Cms.Core.Cache.ObjectCacheAppCache.Get(String key, Func`1 factory, Nullable`1 timeout, Boolean isSliding, String[] dependentFiles)
   at Umbraco.Cms.Core.Cache.ObjectCacheAppCache.Get(String key, Func`1 factory)
   at Umbraco.Cms.Core.Cache.DeepCloneAppCache.Get(String key, Func`1 factory)
   at Umbraco.Extensions.AppCacheExtensions.GetCacheItem[T](IAppCache provider, String cacheKey, Func`1 getCacheItem)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.DocumentRepository.RecycleBinSmells()
   at Umbraco.Cms.Core.Services.ContentService.RecycleBinSmells()
   at Umbraco.Cms.Web.BackOffice.Trees.ContentTreeController.get_RecycleBinSmells()
   at Umbraco.Cms.Web.BackOffice.Trees.ContentTreeControllerBase.GetTreeNodes(String id, FormCollection queryStrings)
   at Umbraco.Cms.Web.BackOffice.Trees.TreeControllerBase.GetTreeNodesAsync(String id, FormCollection queryStrings)
   at Umbraco.Cms.Web.BackOffice.Trees.TreeControllerBase.GetNodes(String id, FormCollection queryStrings)
   at Umbraco.Cms.Web.BackOffice.Trees.ApplicationTreeController.GetChildren(Tree tree, Int32 id, FormCollection querystring)
   at Umbraco.Cms.Web.BackOffice.Trees.ApplicationTreeController.GetTreeRootNode(Tree tree, Int32 id, FormCollection querystring)
   at Umbraco.Cms.Web.BackOffice.Trees.ApplicationTreeController.GetApplicationTrees(String application, String tree, FormCollection queryStrings, TreeUse use)
   at lambda_method2797(Closure, Object)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

I followed the stacktrace and that led me to this query taking "too long":

SELECT value 
FROM umbracoLock WITH (REPEATABLEREAD)  
WHERE id=@id

Where @id is -333 in this case. When I execute this query manually it does not complete within at least 2 minutes (I stopped trying after that). When I execute it without the WITH (REPEATABLEREAD) I do get a result instantaneously.

[matteus-hemstrom-nltg]

We are also still having this issue on 12.3.1. However, the more pressing issue for us is that the locks don't get released in some scenarios, see https://github.com/umbraco/Umbraco-CMS/issues/13804#issuecomment-1816551021.

[bergmania]

@matteus-hemstrom-nltg, I have a feeling the two issues are connected..

This bug report just shows the symptom, that the table is locked. I wrote a potential fix on https://github.com/umbraco/Umbraco-CMS/issues/13804#issuecomment-1840366547, and it would be very helpful if anyone can try it out, to see if it fixes the issue.

[jorgecostadept]

Here is a lineage from the log of 4 items when I attempted to refresh the Internal Index and getting write lock, all on the same process (69):

I recently had this same issue happening "out of nowhere" and by iterating through my commits i found out what caused it. In my specific case, i'm using Hangfire for some scheduled jobs, and i also have an extra custom SQL DB i need to use. For some reason, adding my DBContext before Hangfire was causing the issue in the image above, and after that it would immediately lock the Umbraco DB.

This is a sample of my code:

When the code under "// DB repositories configurations" was before the "// Recurring jobs" code, i would get the issue. once i changed the order, it all started working perfectly.

This is the code from the "ConfigureSqlDbRepositories" and "ApplyMigrations" methods:

I tested this many times, always starting with a new fresh DB and constantly had the same results.

I don't know the root cause yet, but it could be that something in your DI is conflicting with Umbraco and causing these issues. If anything, I hope this helps someone get to a solution :)

[jerpenol]

Any updates on this?

[remkovb]

Any updates on this?

I'd like to know too.. we are stuck with this for a while now.. When doing anything with nodes: publishing, saving, deleting, this error appears a lot:

[dotnetshadow]

We are currently on v10.4.0 and experiencing the same issue for a long time. Looking at all the release notes since then, there doesn't seem to be any fixes that address this issue.

Is this still happening on v13.1.0?