Umbraco.Media throws exception when the website is hosted on Microsoft Azure - An unexpected network error occurred.

mehrdadab commented 5 years ago

We have multiple Umbraco websites hosted on Azure Web Apps. Occasionally we encounter an issue so that everywhere on the website, Umbraco.Media() function throws exception. If we restart the website, the issue disappears.

Reproduction

We cannot reproduce it as it is happening occasionally and only happens on Azure.

Bug summary

This issue happens randomly, When we have 6 websites hosted on the Azure, each time only some of them have that issue. By restarting a website we resolve the issue. Our media hosted on the Azure blob. (I am not sure if it is relevant because I believe Umbraco.Media() reads the URL from a database) and during that issue we have access to blob. For the database, we also use Azure SQL Server database. We don't use load-balancer or auto-scaling.

Specifics

Our Umbraco version is: 7.5.7 and assembly: 1.0.6219.11990 I checked it on Chrome and IE but I believe it is happening on any browser as it is a server error. When on log file I couldn't get too much information, I used a profiler to get more information and this is what I received:

System.IO.IOException: An unexpected network error occurred.
at System.IO.__Error.WinIOError
at System.IO.FileStream.ReadCore
at System.IO.FileStream.Read
at SimpleFSIndexInput.ReadInternal
at Lucene.Net.Store.BufferedIndexInput.ReadBytes
at CSIndexInput.ReadInternal
at Lucene.Net.Store.BufferedIndexInput.Refill
at Lucene.Net.Store.BufferedIndexInput.ReadByte
at Lucene.Net.Store.IndexInput.ReadVInt
at Lucene.Net.Index.TermBuffer.Read
at Lucene.Net.Index.SegmentTermEnum.Next
at Lucene.Net.Index.SegmentTermEnum.ScanTo
at Lucene.Net.Index.TermInfosReader.Get
at Lucene.Net.Index.TermInfosReader.Terms
at MultiTermEnum..ctor
at Lucene.Net.Index.DirectoryReader.Terms
at Lucene.Net.Search.WildcardTermEnum..ctor
at Lucene.Net.Search.WildcardQuery.GetEnum
at ConstantScoreAutoRewrite.Rewrite
at Lucene.Net.Search.WildcardQuery.Rewrite
at Lucene.Net.Search.BooleanQuery.Rewrite
at Lucene.Net.Search.BooleanQuery.Rewrite
at Lucene.Net.Search.IndexSearcher.Rewrite
at Lucene.Net.Search.Query.Weight
at Lucene.Net.Search.Searcher.Search
at Examine.LuceneEngine.SearchResults.DoSearch
at Examine.LuceneEngine.Providers.BaseLuceneSearcher.Search
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMediaCacheValues
at <>c__DisplayClass1.<GetSafeLazy>b__0
at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem
at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem
at Umbraco.Core.Cache.DeepCloneRuntimeCacheProvider.GetCacheItem
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetCacheValues
at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia
at Umbraco.Web.PublishedCache.ContextualPublishedCache`1.GetById
at Umbraco.Web.PublishedContentQuery.Media
at ASP._Page_App_Plugins_MyBlock_Views_HomePage_SearchBox_cshtml.Execute

Also I noticed when this issue is happening if I login to the backoffice and go to the developer tab, I see the error message below on the right hand pannel and don't have access to Examine management:

Received an error from the server
An error occured
The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.

EXCEPTION DETAILS
System.InvalidOperationException: The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.
STACKTRACE
INNER EXCEPTION
System.Net.Http.HttpRequestException: Error while copying content to a stream.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.WebHost.HttpControllerHandler.<WriteBufferedResponseContentAsync>d__1b.MoveNext()
INNER EXCEPTION
System.IO.IOException: An unexpected network error occurred.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.ReadCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at Lucene.Net.Store.SimpleFSDirectory.SimpleFSIndexInput.ReadInternal(Byte[] b, Int32 offset, Int32 len)
   at Lucene.Net.Store.BufferedIndexInput.Refill()
   at Lucene.Net.Store.BufferedIndexInput.ReadByte()
   at Lucene.Net.Store.IndexInput.ReadVInt()
   at Lucene.Net.Index.CompoundFileReader..ctor(Directory dir, String name, Int32 readBufferSize)
   at Lucene.Net.Index.SegmentReader.CoreReaders..ctor(SegmentReader origInstance, Directory dir, SegmentInfo si, Int32 readBufferSize, Int32 termsIndexDivisor)
   at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, SegmentInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDivisor)
   at Lucene.Net.Index.IndexWriter.ReaderPool.Get(SegmentInfo info, Boolean doOpenStores, Int32 readBufferSize, Int32 termsIndexDivisor)
   at Lucene.Net.Index.IndexWriter.ReaderPool.GetReadOnlyClone(SegmentInfo info, Boolean doOpenStores, Int32 termInfosIndexDivisor)
   at Lucene.Net.Index.DirectoryReader..ctor(IndexWriter writer, SegmentInfos infos, Int32 termInfosIndexDivisor)
   at Lucene.Net.Index.IndexWriter.GetReader(Int32 termInfosIndexDivisor)
   at Umbraco.Web.Search.ExamineExtensions.GetIndexDocumentCount(LuceneIndexer indexer)
   at Umbraco.Web.WebServices.ExamineManagementApiController.CreateModel(BaseIndexProvider indexer)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
   at System.Net.Http.Formatting.JsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStreamAsync(Type type, Object value, Stream writeStream, HttpContent content, TransportContext transportContext, CancellationToken cancellationToken)

Steps to reproduce

We couldn't reproduce it on our local environment at all. Also we have same kind of website on an Azure Server in Australia and we never had the issue. It seems, it only happens on our Uk server. In addition, you can see on both error messages Lucene.net has been mentioned, that's why I think it is related to indexing.

Expected result

We call Umbraco.Media to get our Media Url .

Actual result

Umbraco.Media throws an exception, so we don't have URL of media and for example our images are blank.

nul800sebastiaan commented 5 years ago

Hey there @mehrdadab - Unfortunately you're using a pretty old version of Umbraco and your problem seems related to the specific server environment. If you have any way of reproducing this on the latest version of Umbraco we could have a look at it but hopefully this problem has been fixed along the way already.

I would recommend you try and get some help on the forum to see if anyone else experienced this and has a possible solution: https://our.umbraco.com/

mehrdadab commented 5 years ago

FYI, I upgraded the website to Umbraco 7.13.1 and I have still the same issue. I cannot reproduce it though, as it happens randomly on Microsoft Azure. I am still working on it and let you know if I found any clue.

mehrdadab commented 5 years ago

I also found out unlike umbraco 7.5.7 on umbraco 7.13.1 I have access to Examine Management and if I rebuild index, the problem disappears.

maarten-kieft commented 5 years ago

@nul800sebastiaan , is it possible to reopen the issue? As @mehrdadab says he still has the issue and I'm experiencing the same issue with the latest version of Umbraco. To give you a bit more details: It happens from time to time, randomly. It happens on azure, but also locally on windows machine with iis.