azure-contrib / AzureDirectory

A Lucene Directory Provider for Azure Blob Storage
Microsoft Public License
77 stars 57 forks source link

Speed of adding document to azure directory. #2

Open pe4aiko opened 11 years ago

pe4aiko commented 11 years ago

Hi. I have some code: var cloudAccount = CloudStorageAccount.Parse("someAccount"); var cacheDirectory = new RAMDirectory(); var azureDirectory = new AzureDirectory(cloudAccount, "fulltextcloudindex", cacheDirectory); Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);

var indexWriter = new IndexWriter(azureDirectory, analyzer, true, IndexWriter.MaxFieldLength.UNLIMITED); var luceneDataProvider = new LuceneDataProvider(azureDirectory, Version.LUCENE_30, IndexWriter);

using (var sessionArticle = LuceneDataProvider.OpenSession()) { sessionArticle.Add(entity); }

When I try to step out from using.. it gets about 10 seconds. It's really slow. Or maybe I need to do it in another way?

The output: COMPRESSED 201 -> 168 83.58209% to _2g.fdt PUT 168 bytes to _2g.fdt in cloud CLOSED WRITESTREAM _2g.fdt COMPRESSED 12 -> 9 75% to _2g.fdx PUT 9 bytes to _2g.fdx in cloud CLOSED WRITESTREAM _2g.fdx COMPRESSED 348 -> 230 66.09196% to _2g.tis PUT 230 bytes to _2g.tis in cloud CLOSED WRITESTREAM _2g.tis COMPRESSED 35 -> 26 74.28571% to _2g.tii PUT 26 bytes to _2g.tii in cloud CLOSED WRITESTREAM _2g.tii COMPRESSED 28 -> 12 42.85714% to _2g.frq AzureLock:Renew(write.lock : fe28a1f6-2597-4383-85f8-5efe27c23534 PUT 12 bytes to _2g.frq in cloud CLOSED WRITESTREAM _2g.frq COMPRESSED 20 -> 17 85% to _2g.prx PUT 17 bytes to _2g.prx in cloud CLOSED WRITESTREAM _2g.prx COMPRESSED 16 -> 12 75% to _2g.nrm PUT 12 bytes to _2g.nrm in cloud CLOSED WRITESTREAM _2g.nrm PUT 155 bytes to _2g.fnm in cloud CLOSED WRITESTREAM _2g.fnm opening _2g.fdt Using cached file for _2g.fdt CLOSED READSTREAM local _2g.fdt opening _2g.fdx Using cached file for _2g.fdx CLOSED READSTREAM local _2g.fdx opening _2g.tis Using cached file for _2g.tis CLOSED READSTREAM local _2g.tis opening _2g.tii Using cached file for _2g.tii CLOSED READSTREAM local _2g.tii opening _2g.frq Using cached file for _2g.frq CLOSED READSTREAM local _2g.frq opening _2g.prx Using cached file for _2g.prx CLOSED READSTREAM local _2g.prx opening _2g.nrm Using cached file for _2g.nrm CLOSED READSTREAM local _2g.nrm opening _2g.fnm Using cached file for _2g.fnm CLOSED READSTREAM local _2g.fnm COMPRESSED 944 -> 528 55.93221% to _2g.cfs PUT 528 bytes to _2g.cfs in cloud CLOSED WRITESTREAM _2g.cfs DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.fnm DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.frq DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.prx DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.tis DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.tii DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.nrm DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.fdx DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2g.fdt opening _2g.cfs Using cached file for _2g.cfs Creating clone for _2g.cfs CLOSED READSTREAM local Creating clone for _2g.cfs Creating clone for _2g.cfs CLOSED READSTREAM local Creating clone for _2g.cfs Creating clone for _2g.cfs Creating clone for _2g.cfs Creating clone for CLOSED READSTREAM local Creating clone for Creating clone for CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local _2g.cfs AzureLock:Renew(write.lock : fe28a1f6-2597-4383-85f8-5efe27c23534 PUT 212 bytes to segments_2x in cloud CLOSED WRITESTREAM segments_2x PUT 20 bytes to segments.gen in cloud CLOSED WRITESTREAM segments.gen DELETE https://company.blob.core.windows.net/fulltextcloudindex/segments_2w DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2d.cfs DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2d_1.del DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2e.cfs DELETE https://company.blob.core.windows.net/fulltextcloudindex/_2f.cfs opening segments.gen Using cached file for segments.gen CLOSED READSTREAM local segments.gen opening segments_2x Using cached file for segments_2x CLOSED READSTREAM local segments_2x opening _2g.cfs Using cached file for _2g.cfs Creating clone for _2g.cfs CLOSED READSTREAM local Creating clone for _2g.cfs Creating clone for _2g.cfs CLOSED READSTREAM local Creating clone for _2g.cfs Creating clone for _2g.cfs Creating clone for _2g.cfs Creating clone for _2g.cfs Creating clone for Creating clone for Creating clone for _2g.cfs CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local _2d.cfs CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local _2e.cfs CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local CLOSED READSTREAM local _2f.cfs

richorama commented 11 years ago

Sorry, I just ported the code to the new storage library, so I don't really have the authority to talk about performance expectations.

Just out of interest, what size blobs did it create? Are you running this inside the same data centre as the storage account?