umbraco / Umbraco-CMS

Umbraco is a free and open source .NET content management system helping you deliver delightful digital experiences.
https://umbraco.com
MIT License
4.49k stars 2.69k forks source link

Fails to load NuCache #12213

Closed bjarnef closed 2 years ago

bjarnef commented 2 years ago

Which exact Umbraco version are you using? For example: 9.0.1 - don't just write v9

9.4.1

Bug summary

Not sure if this is related to https://github.com/umbraco/Umbraco-CMS/pull/12209 but after upgrading from 9.2.0 to 9.4.1 it seems we get some errors loading NuCache.

We are importing some store nodes using Hangfire, which we tested on 9.2.0, but it seems after upgrading to 9.4.1 we get some errors as the Hangfire job fails with this exception.

System.AggregateException: Exceptions were thrown by listed actions. (An unexpected network error occurred. : 'C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\NuCache.Content.db')
 ---> System.IO.IOException: An unexpected network error occurred. : 'C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\NuCache.Content.db'
   at System.IO.FileStream.ReadNative(Span`1 buffer)
   at System.IO.FileStream.ReadSpan(Span`1 destination)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at CSharpTest.Net.IO.StreamCache.CachedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at CSharpTest.Net.IO.IOStream.ReadChunk(Stream io, Byte[] bytes, Int32 offset, Int32 length)
   at CSharpTest.Net.IO.TransactedCompoundFile.ReadBytes(Int64 position, Byte[] bytes, Int32 offset, Int32 length)
   at CSharpTest.Net.IO.TransactedCompoundFile.FileSection.Read(BlockRef& block, Boolean headerOnly, FGet fget)
   at CSharpTest.Net.IO.TransactedCompoundFile.Read(UInt32 handle)
   at CSharpTest.Net.Storage.BTreeFileStoreV2.TryGetNode[TNode](IStorageHandle handleIn, TNode& node, ISerializer`1 serializer)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone.Lock(NodePin parent, LockType ltype, NodeHandle child)
   at CSharpTest.Net.Collections.BPlusTree`2.RootLock..ctor(BPlusTree`2 tree, LockType type, Boolean exclusiveTreeAccess, String methodName)
   at CSharpTest.Net.Collections.BPlusTree`2.AddEntry[T](TKey key, T& info)
   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.Release(WriteLockInfo lockInfo, Boolean commit)
   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.ScopedWriteLock.Release(Boolean completed)
   at Umbraco.Cms.Core.Scoping.ScopeContextualBase.<>c__1`1.<Get>b__1_1(Boolean completed, T item)
   at Umbraco.Cms.Core.Scoping.ScopeContext.EnlistedObject`1.Execute(Boolean completed)
   at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
   --- End of inner exception stack trace ---
   at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
   at Umbraco.Cms.Core.Scoping.Scope.<>c__DisplayClass106_0.<RobustExit>b__2()
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)
   at Umbraco.Cms.Core.Scoping.Scope.DisposeLastScope()
   at Umbraco.Cms.Core.Scoping.Scope.Dispose()
   at Umbraco.Cms.Core.Services.Implement.ContentService.SaveAndPublish(IContent content, String culture, Int32 userId)

I have simplified the code a bit:

public interface IStoreSyncHandler
{
    [AutomaticRetry(Attempts = 0, OnAttemptsExceeded = AttemptsExceededAction.Fail)]
    [JobDisplayName("Sync stores")]
    public Task RunAsync(PerformContext context);
}

public class StoreSyncHandler : IStoreSyncHandler
{
    private readonly ILogger<StoreSyncHandler> _logger;

    private readonly IUmbracoContextFactory _umbracoContextFactory;
    private readonly IPublishedSnapshotAccessor _publishedSnapshotAccessor;
    private readonly IContentService _contentService;
    private readonly IIOHelper _ioHelper;
    private readonly MediaFileManager _mediaFileManager;
    private readonly GoogleGeocoder _geocoder;

    public StoreSyncHandler(
        ILogger<StoreSyncHandler> logger,
        IUmbracoContextFactory umbracoContextFactory,
        IPublishedSnapshotAccessor publishedSnapshotAccessor,
        IContentService contentService,
        IIOHelper ioHelper,
        MediaFileManager mediaFileManager,
        IOptions<GoogleMapsConfig> config)
    {
        _logger = logger;
        _umbracoContextFactory = umbracoContextFactory;
        _publishedSnapshotAccessor = publishedSnapshotAccessor;
        _contentService = contentService;
        _ioHelper = ioHelper;
        _mediaFileManager = mediaFileManager;

        _geocoder = new GoogleGeocoder(config.Value.ApiKey);
    }

    public async Task RunAsync(PerformContext context)
    {
        try
        {
            using (var cref = _umbracoContextFactory.EnsureUmbracoContext())
            {
                var cache = cref.UmbracoContext.Content;

                var root = cache.GetAtRoot().FirstOrDefault();
                if (root == null)
                    throw new Exception("No root was found.");

                var settings = root.FirstChild<ContentModels.Settings>();
                var storeArchive = root.FirstChild<ContentModels.StoreLocator>();

                if (storeArchive == null)
                    throw new Exception("No store archive was found.");

                var importFile = settings?.StoreImportFile;

                if (string.IsNullOrEmpty(importFile))
                {
                    await Task.CompletedTask;
                }

                if (!importFile.EndsWith(".csv", StringComparison.InvariantCultureIgnoreCase))
                    throw new Exception("Import file is not a CSV file.");

                var config = new CsvConfiguration(CultureInfo.InvariantCulture)
                {
                    Delimiter = ",",
                    DetectColumnCountChanges = true,
                    HasHeaderRecord = true,
                    HeaderValidated = null,
                    MissingFieldFound = null,
                    IgnoreBlankLines = false,
                    Mode = CsvMode.RFC4180,
                    // Ignore header case.
                    PrepareHeaderForMatch = args => args.Header.ToLower()
                };

                string path = _ioHelper.GetRelativePath(importFile);

                path = path.TrimStart(Constants.CharArrays.Tilde);

                context.WriteLine(ConsoleTextColor.Gray, $"CSV path: {path}");

                if (_mediaFileManager.FileSystem.FileExists(path))
                {
                    var stream = _mediaFileManager.FileSystem.OpenFile(path);
                    stream.Seek(0, SeekOrigin.Begin);

                    using (var reader = new StreamReader(stream))
                    using (var csv = new CsvReader(reader, config))
                    {
                        var records = new List<StoreRecord>();

                        try
                        {
                            while (csv.Read())
                            {
                                try
                                {
                                    var record = csv.GetRecord<StoreRecord>();
                                    records.Add(record);
                                }
                                catch (TypeConverterException ex)
                                {
                                    _logger.LogError(ex, "Failed reading record.");
                                }
                                catch (Exception ex)
                                {
                                    _logger.LogError(ex, "Failed reading record.");
                                }
                            }

                            // Unique records
                            records = records.Where(x => x.CustomerNumber.HasValue && x.CustomerNumber.Value > 0)
                                                .DistinctBy(x => x.CustomerNumber.Value)
                                                .ToList();

                            var customerNumbers = records.Select(x => x.CustomerNumber).ToArray();
                            var existingStores = new List<StoreRecord>();

                            const int pageSize = 500;
                            var page = 0;
                            var total = long.MaxValue;

                            while (page * pageSize < total)
                            {
                                var children = _contentService.GetPagedChildren(storeArchive.Id, page++, pageSize, out total);

                                foreach (var child in children)
                                {
                                    var customerNumber = child.GetValue<int?>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.CustomerNumber).Alias);

                                    if (customerNumber == null)
                                        continue;

                                    // Check if exists
                                    var found = records.FirstOrDefault(x => x.CustomerNumber == customerNumber);
                                    if (found == null)
                                        continue;

                                    existingStores.Add(found);

                                    await CreateOrUpdateNode(storeArchive.Key, child, found);
                                }
                            }

                            var newStores = records.Except(existingStores) ?? Enumerable.Empty<StoreRecord>();
                            var totalNewStores = newStores.Count();

                            foreach (var store in newStores)
                            {
                                await CreateOrUpdateNode(storeArchive.Key, null, store);
                            }
                        }
                        catch (BadDataException ex)
                        {
                            _logger.LogError(ex, "Failed reading CSV.");

                            throw;
                        }
                        catch (Exception ex)
                        {
                            _logger.LogError(ex, "Failed reading CSV.");

                            throw;
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {
            var message = e.Message ?? string.Empty;

            _logger.LogError(e, "Exception: {0} | Message: {1} | Stack Trace: {2}", e.InnerException != null ? e.InnerException.ToString() : "", message, e.StackTrace);

            throw;
        }
    }

    private async Task CreateOrUpdateNode(Guid parentId, IContent node, StoreRecord store)
    {
        if (store == null)
            return;

        string statusValue = store.Status?.Trim();
        statusValue = statusValue?.Length >= 1 ? statusValue.Substring(0, 1) : null;

        GoogleAddress googleAddress = null;

        string title = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Title).Alias);
        string location = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Location).Alias);
        string address = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Address).Alias);
        string zip = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Zip).Alias);
        string city = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.City).Alias);
        string country = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Country).Alias);
        string email = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Email).Alias);
        string phone = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Phone).Alias);

        var status = node?.GetValue<PartnerStatus>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Status).Alias);
        var partnerStatus = statusValue == null ? (PartnerStatus?)null : Enum.TryParse(statusValue, true, out PartnerStatus ps) ? ps : default;

        bool isNew = node == null;

        bool isDiffStatus = status != partnerStatus;

        int diffCount = 0;

        bool isDiffAddress = (Comparison.AreEqual(store.Address, address, ref diffCount) &&
                            Comparison.AreEqual(store.Zip, zip, ref diffCount) &&
                            Comparison.AreEqual(store.City, city, ref diffCount) &&
                            Comparison.AreEqual(store.CountryName, country, ref diffCount)) == false;

        bool isDiffProps = isDiffAddress || isDiffStatus || (Comparison.AreEqual(store.Name, title, ref diffCount) && Comparison.AreEqual(store.Email, email, ref diffCount) && Comparison.AreEqual(store.Phone, phone, ref diffCount)) == false;

        // If not new entry or values of properties hasn't changed we don't need to continue
        if (!isNew && !isDiffProps)
            return;

        // Lookup address if location not is set or address has changed
        if (string.IsNullOrEmpty(location) || isDiffAddress)
        {
            googleAddress = await GetAddress(store.Address, store.Zip, store.City, store.CountryName);
        }

        var name = $"{store.CustomerNumber} - {store.Name}";

        node ??= _contentService.Create(name, parentId, ContentModels.Store.ModelTypeAlias);

        node.Name = name;

        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.CustomerNumber).Alias, store.CustomerNumber);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Title).Alias, store.Name);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Address).Alias, store.Address);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Zip).Alias, store.Zip);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.City).Alias, store.City);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Country).Alias, store.CountryName);

        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Email).Alias, store.Email);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Phone).Alias, store.Phone);

        if (partnerStatus != null)
        {
            node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Status).Alias, partnerStatus);
        }

        if (googleAddress != null)
        {
            var map = new Our.Umbraco.GMaps.Models.Map();

            map.Address.Coordinates = new Our.Umbraco.GMaps.Models.Location
            {
                Latitude = googleAddress.Coordinates.Latitude,
                Longitude = googleAddress.Coordinates.Longitude
            };

            map.Address.FullAddress = googleAddress.FormattedAddress;
            map.Address.PostalCode = store.Zip;
            map.Address.City = store.City;
            map.Address.State = "";
            map.Address.Country = store.CountryName;

            var mapJson = Newtonsoft.Json.JsonConvert.SerializeObject(map);

            // Store JSON in GMaps property editor
            node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Location).Alias, mapJson);
        }

        _contentService.SaveAndPublish(node);
    }

    private async Task<GoogleAddress> GetAddress(string address, string zip, string city, string country)
    {
        try
        {
            var results = await _geocoder.GeocodeAsync(address, city, "", zip, country);
            var result = results.FirstOrDefault(x => !x.IsPartialMatch);

            if (result == null || result.Coordinates == null)
                return null;

            return result;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"Failed to lookup address: {address}, {zip} {city}, {country}.");
        }

        return null;
    }
}

Furthermore when reloading memory cache from Published Status dashboard it fails.

image

Also when saving and publish a content node it fails.

image

The site is hosted on Azure Web Apps, but not sure if that makes a difference.

Specifics

No response

Steps to reproduce

Import some content nodes via Hangfire and notice the NuCache may be corrupt. It may require a larger amount of data to reproduce.

In this specific project they have 7347 store nodes.

Expected result / actual result

No response

nzdev commented 2 years ago

Hi @bjarnef, how many umbraco instances are involved? Have you seen this blog on using hangfire and umbraco content api. You may in addition want to check IMainDom to check the current process has acquired maindom.

bjarnef commented 2 years ago

Hi @nzdev The Azure environment has one Umbraco instance, but two slots for "staging" and "production". But it shouldn't differ from other of our Azure hosted Umbraco projects. However this it the first one on v9 using Hangfire. Not sure if that makes a difference.

In fact we have pretty much same import logic on a similar project it v8 using Hangfire, where I haven't seen this error.

nzdev commented 2 years ago

I've had similar code on V8 as well and needed to use the ScopeProvider and BackgroundScope as well per that blog post when it was converted to V9.

For this issue, perhaps set MainDom lock to FileSystemMainDomLock per https://github.com/umbraco/Umbraco-CMS/pull/12037 and check LocalTempStorageLocation is set to EnvironmentTemp as C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\NuCache.Content.db' is going to be on the network drive on Azure which won't work as both staging and production will be writing the same file, which is not supported, as this is the root cause of the issue. Perhaps an Azure Web Apps config health check would help in future.

bjarnef commented 2 years ago

Actually I tried using ScopeProvider in v8 here, but for some reason it was throwing an exception when I used that: https://github.com/umbraco/Umbraco-CMS/issues/10908#issuecomment-903934643

I could try with the ScopeProvider or BackgroundScope, but it didn't seem to be an issue importing content, but mainly to update or build NuCache.

nzdev commented 2 years ago

Order of the scope and umbraco context using statements matter. Umbraco context should be disposed after the IScope.

bjarnef commented 2 years ago

@nzdev we tried with FileSystemMainDomLock but the NuCache still seemed to be broken 🙈 Locally I haven't any issues though, so I guess it is Azure related.

I modified the import logic to use the suggested approach in the blogpost you mentioned https://github.com/umbraco/Umbraco-CMS/issues/12213#issuecomment-1087150192

using var backgroundScope = new BackgroundScope(_serverMessenger);
using var _ = _umbracoContextFactory.EnsureUmbracoContext();
using var serviceScope = _serviceProvider.CreateScope();

var cache = serviceScope.ServiceProvider.GetRequiredService<IPublishedContentQuery>();

...
nzdev commented 2 years ago

Have you deleted the nucache.db files so they are regenerated.

p-m-j commented 2 years ago

check LocalTempStorageLocation is set to EnvironmentTemp

This is really important for running on azure app services.

bjarnef commented 2 years ago

@nzdev okay, we can try deleting the NuCache.db

@p-m-j I believe we already have LocalTempStorageLocation set to EnvironmentTemp, but I will check this as well.

We use the recommended configuration as mentioned in the documentation here: https://our.umbraco.com/Documentation/Fundamentals/Setup/server-setup/azure-web-apps#recommended-configuration

image

nzdev commented 2 years ago

@bjarnef it looks like you are using app service configuration. Replace : with __ for the keys.

p-m-j commented 2 years ago

: is fine for Windows app services but dunder __ works x-plat

nzdev commented 2 years ago

@p-m-j : didn't seem to work for me on windows.

p-m-j commented 2 years ago

@p-m-j I believe we already have LocalTempStorageLocation set to EnvironmentTemp, but I will check this as well.

Did you get to the bottom of this? With EnvironmentTemp configured NuCache.Content.db should be found at C:\local\Temp\UmbracoData\{appid-hash}\NuCache\ but your stacktrace references C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\

bjarnef commented 2 years ago

@p-m-j after I changed to the following it seems to work.

using var backgroundScope = new BackgroundScope(_serverMessenger);
using var _ = _umbracoContextFactory.EnsureUmbracoContext();
using var serviceScope = _serviceProvider.CreateScope();

var cache = serviceScope.ServiceProvider.GetRequiredService<IPublishedContentQuery>();

Strange enough it seemed to work fine without the serviceScope in v8.

In v8 I tried with the CreateScope() but I got some errors at that time - it did however seem to work without the wrapping scope: https://github.com/umbraco/Umbraco-CMS/issues/10908#issuecomment-903934643

p-m-j commented 2 years ago

@bjarnef glad you’re sorted, can this issue be closed then?

rossparachute commented 4 months ago

This was the only fix that worked for us.

Might be an Azure config reason we can't yet get it working with LocalTempStorageLocation set to EnvironmentTemp. Looks like the data never gets created in the Azure WebApp container under: _C:\local\Temp\UmbracoData\uniqueid\NuCache\NuCache.Content.db

Unsure if Kudu has access to that file but it doesn't look like there's anything there. Will post in if we sort this on legacy app.

Haven't found this to be an issue in Umbraco v11+.