slow calculation

[Artpom]

From latest release i saw that some calculations scheduled for 5minutes takes 40minutes or even longer. For no reason sometimes calculation process slows down to such level that whole process is getting longer by 30-40minutes. Sometimes it happens to two threads at the same time. I have set max threads for 5, so if 2 of those practically do not progress then whole creation process is getting horribly long. With v 2.06 i haven't seen such behaviour.

[stackTom]

Hi, can you send me your partialfsearthtiles.ini for an area exhibiting this behavior so I can troubleshoot? Thanks

[Artpom]

Hmm... i do not have such file Here you have my config file:

[Service1] ServiceName = Virtual Earth ServiceCodeing = 0123 ServiceUrl = http://a1.ortho.tiles.virtualearth.net/tiles/a%s.jpeg?g=172 #you need to replace the * with a service base url

Referer = http://a1.ortho.tiles.virtualearth.net #optional (older versions g=104, g=52)

UserAgent = Mozilla/4.0 #optional

ServerVariations = a0,a1,a2,a3 #Server variations that all have to contain the same Tiles

[Service2] ServiceName = Google Earth ServiceCodeing = xyinvz ServiceUrl =http://mt0.google.com/vt/lyrs=s&%s Referer = http://mt0.google.com UserAgent = Mozilla/4.0 ServerVariations = mt0,mt1,mt2,mt3

[ProxyList] #if you want to use proxies (web/http) add them here direct #use the word "direct" in the List to mark a connection without Proxy

127.0.0.1:8118 #local proxy (example for TOR net)

130.149.49.26:3124 #external proxy example

direct #you may enter the same Proxies more than one time in this list if you like

132.239.17.224:68664 #Use the next button to change cycle throughthe proxy list manualy. (note original I planed a proxy change automatisation but didn't implement it at the end)

---FS Earth Tiles Start up default values---

[FSEarthTiles]

WorkingFolder = E:\FSET\working #working space folder. SceneryFolder = F:\Addon Scenery #scenery folder. Note it stores the files in the texture and scenery Sub-folder (also creates any missing folders)

UseCache = Yes BlankTileColorRed = 0 BlankTileColorGreen = 30 BlankTileColorBlue = 35 DownloadResolution = 5 #-4...8 StartWithService = Google Earth SelectedSceneryCompiler = FSX/P3D #choose: FSX or FS2004
AreaSnap = Tiles #Off/LOD13/LatLong/Tiles/Pixel, IMPORTANT: Select LOD13 for FS2004 to avoid water borders! CreateAreaMaskBmp = Yes #if Yes Area Mask Bitmap will be created that are used for Blend and Water and Seasons. (note FS2004 does not know blend) CompileScenery = Yes #Yes = Compile the Scenery, No = Download the Area only. AutoReferenceMode = 1 # 1..4 only active if AreaSnap is not off

1 = match exact Area Dimensions, 2 = allow to go one large, 3 = maximum freedom in going larger (largest areas simular to Area Snap Off), 4 = divide into single units

Zoom = 12 # -4...18

AreaDefinitionMode = 1Point #1Point / 2Points AreaSizeX = 1 #[nm] Usage of this value depends on your AreaDefinitionMode setting (used on 1Point) AreaSizeY = 0.6 #[nm] Use point for broken numbers here such as 1.5 or 0.75 no matter what language setting you use CenterLatitude = 26deg 25min 00sec north #Usage of this value depends on your AreaDefinitionMode setting (used on 1Point) CenterLongitude = 80deg 51min 00sec west NorthWestCornerLatitude = 44deg 26min 00sec north #Usage of this value depends on your AreaDefinitionMode setting (used on 2Points) NorthWestCornerLongitude = 8deg 50min 00sec east SouthEastLatitude = 44deg 25min 00sec north #Usage of this value depends on your AreaDefinitionMode setting (used on 2Points) SouthEastLongitude = 8deg 51min 00sec east

-- Additional Settings --

TextureUndistortion = Perfect # choose: Off/Good/Perfect/PerfectHighQualityFSPreResampling. It corrects the texture to ensure accuracy on large areas and in best mode it heavy improves FSCompiler result but costs a lot memory

Off -> Not recommended (inaccurate on large areas)

                                                             # Good    -> Fast with Good Quality, small Memory usage 
                                                             # Perfect -> Remarkable slower but perfect accuracy, small Memory usage 
                                                             # PerfectHighQualityFSPreResampling -> Increase of Texture Quality through Pre-Resampling (Doing FS's job, just with much more quality result). Heavy Memory usage (6 times more). As slows as Undistortion Perfect.

FSPreResampleingAllowShrinkTexture = Yes # Yes = Shrink the Texture on resampling instead enlarging. That generates one LOD Level less. Saves HD and memory space. Sharper but less Antialiasing effect. (only aktive if TextureUndistortion = PerfectHighQualityFSPreResampling) FSPreResampleingShrinkTextureThresholdResolutionLevel = 3 # Textures with selected Resolution Level less or equal this values becomes shrink instead enlarged. Usefull if HD space is limited to get the maximum quality for the highest LOD.

Example. The Default value 3 means: Dl Res 3 was earlier converted to one LOD up: LOD14, and with this it will be shrinked/converted in LOD13 (which match more with the table on the GUI). Dl Res 4 will still be converted upward to LOD13 (only active if FSPreResampleingAllowShrinkTexture = Yes)

Brightness = -6.0 #[percent] Texture Color Enhancement. Brighness is best set to BrighnessValue = - 0.5 * ContrastValue;
Contrast = +12.0 #[percent] Texture Color Enhancement.

DownloadFailTimeOut = 1800 #Time out in seconds. If connection breaks and the tile flow stops this is the time after which the download attempts stops.

the below three settings need to be adjusted based on the speed of your computer

to produce the optimum rate of scenery creation

MaxResampleThreads = 4 # the amount of resample (and mask creation) threads to use. Minimum of 1 MaxImageProcessingThreads = 4 # the amount of Image Processing threads threads to use. Minimum of 1 MaxDownloadThreads = 8 # the amount of download threads to use. Minimum of 4

MaxMemoryUsageFactor = 1.0 #Factor of Maximum of Memory Usage of detected allocable texture memory. (there is already a 5% marging). WebBrowserRecoverTimer = 15 #[sec] If the WebBrowser Tile flow stops due not accessible pages or redirected or user interruptuions etc the flow is auto restarted after this time

WebBrowserNoTileFoundKeyWords = Not Found,HTTP Error 404,Bad Request #If a WebPage with any of this KeyWords are found (seperated with comma) instead a Tile, The No Tile Found Tile will be used. (else it tries again and again and again..endless..a WebBrowser only problem...(no proper HTTP error msg accessable or I found none)).

Note that the Words used here are Service dependent. Also make sure there is no collision with the Windows internal Page if connection breake. The Wordsyou use should not apear on the Windows connection lost info page else you get No Tile Founds on connection breake also which we don't want.

--- Yes/No Switches --

UseCSharpScripts = Yes #If yes then the Scripts will be loaded and usesd. KeepAreaInfFile = No #if Yes then this keeps the Area.inf File (used for FS scenery compilation with resample.exe with no Water or Season Masks) KeepAreaMaskInfFile = No #if Yes then this keeps the AreaMasks.inf File (used for FS scenery compilation with resample.exe with Water Masks) KeepAreaMaskSeasonInfFile = No #if Yes then this keeps the AreaMasksSeasons.inf File (used for FS scenery compilation with resample.exe with Season Masks (and water)) KeepAreaEarthInfoFile = No #if Yes then this keeps the AreaEarthInfo.txt File (used for Water and Season creation) required input file for FSEarthMasks KeepSourceBitmap = No #if Yes then this keeps the Area.bmp File (This is the original downloaded and assembled Area.bmp and is required for FSEarthMasks and scenery compilation with Area.inf) KeepSummerBitmap = No #if Yes then this keeps the AreaSummer.bmp File (used in AreaMask.inf for scenery compilation with Water/Season Masks. Can be recreated through FSEarthMasks) KeepMaskBitmap = No #if Yes then this keeps the AreaMask.bmp File (used in AreaMask.inf for scenery compilation with Water/Season Masks. Can be recreated through FSEarthMasks) KeepSeasonsBitmaps = No #if Yes then this keeps the Various Season Bitmaps Files (used in AreaMask.inf for scenery compilation with Water/Season Masks. Can be recreated through FSEarthMasks) FS2004KeepTGAs = No #choose: Yes or No (you will find them in the \yourscenery\texture\TgaSourceFiles folder) UseInformativeAreaNames = Yes #No -> Area.bmp, Yes -> Area_Lp2_SnapLatLong_N042470000_N042400000_E010050000_E010170000.bmp ShuffleTilesForDownload = No #if you want the Tiles to be shuffeled downloaded instead in plain order ShuffleAreasForDownload = No #if you want the Areas to be shuffeled downloaded instead in plain order AutoStartDownload = No #Automatical starts the download on application start AutoExitApplication = No #Automatical exits the application after the work is done OpenWebBrowsersOnStart = No #Yes = Right open the WebBrowser Engine on Start of the Application. ShowDisplaySelector = No #Yes = shows the Display mode selector with the 2 modes Free/Tile. Usually of no use except for study/debug.
SuppressPitchBlackPixels = No #Yes = All absolut black Pixels color(0,0,0) will be transformed into color(0x01,0x01,0x01) pixels. CompileWithAreaMask = Yes #Yes = Compiles the Scenery by useing AreaMasks (AreaFS..MasksInfo.inf) . Should best stay on yes. (only active when CreateAreaMaskBmp = Yes) SkipAllWaterTiles = Yes #Yes = Don't create bgl's of tiles which consist solely of water

--- FS compiler ---

UseLODLimits = Yes #Set to Yes to aktivate FSET's minimum destination LOD creation logic. (effects FSX only) MinimumDestinationLOD = 13 #Generates at least LOD's up to this LVL to avoid FS default scenery comeing through. Only active when UseLODLimits = Yes CompressionQuality = 95 #FS default is 100%. Some like 80 better to generate smaller bgl's on cost of teh quality.

--- FS Compilers ---

FSXSceneryCompiler = resampleFSXSP2.exe FS2004SceneryCompiler = resampleFS2004.exe FS2004SceneryImageTool = imagetoolFS2004.exe #if missing the FS2004 compilation stops with the .tga's (no complete FS2004 executable scenery!) MSFS2020SceneryCompiler = fspackagetool.exe

--- FS Earth Masks ---

FSEarthMasksTool = FSEarthMasks.exe

--- Earth Masks Config ---

CreateWaterMaskBitmap = Yes # yes = create Water/Blend Mask. (An Area Vectors file (.Kml or .svg) is required for this!)

CreateSummerBitmap = Yes # yes = create Summer Texture Important! -> , A summer Texture is required when you want to work with FSEM's Color Adaption for Water Regions.

CreateNightBitmap = No # yes = create Night Texture CreateSpringBitmap = Yes # yes = create Spring Texture CreateAutumnBitmap = Yes # yes = create Autumn Texture (Thomas M. routines check FSEM Scripts) CreateWinterBitmap = Yes # yes = create Winter Texture (Thomas M. routines check FSEM Scripts) CreateHardWinterBitmap = No # yes = create Hard Winter Texture (Thomas M. routines check FSEM Scripts)

BlendBorders = Yes # yes = auto blend Borders (Blend-Transition) of an (Multi-)Area

--- Third Party Tools ---

UseAreaKMLFile = No #yes = an AreaKML.kml File is expected in the Work Folder used for Water and Blend Vectors. You need to create that File in advance! UseScalableVectorGraphicsTool = No #yes = Starts the SVG Tool for every Downloaded Area (Only if CreateAreaMaskBmp is on) ScalableVectorGraphicsTool = D:\Inkscape\inkscape.exe #Used for Drawing Water and Blend Vectors creating a .SVG File

--- Scenproc ---

scenproc_loc = E:\FSET\scenProc_3.0_x64\scenProc.exe FS9_scenproc_script = FS9.spc FSX_P3D_scenproc_script = FSXP3D.spc CreateScenproc = Yes

[stackTom]

Thanks, please go to an area that is having these slow downs. Then press the little E button on the top right. This will produce a file called PartialFSEarthTiles.ini in your work folder. Please paste that file here once you have it.

This file will allow me to visit the same area in my fsearthtiles and see if these slow downs are a new bug introduced in version 2.08.

[Artpom]

OK, next time will do if slow down occurs. As a frequent user i have few things to share, and some requests for future updates, how can i share those with you - if you are interested of course

[stackTom]

Sounds good. Feel free to leave another issue here on GitHub and tag it with feature request 👍.

[Artpom]

By the way - area with slow processing was New Zealand - the project was to create entire island ( +1200 tiles )

[stackTom]

How long does it take? And is it any faster with versions before 2.0.8? On my system, 2.0.8 is the fastest.

[Artpom]

I have not tried to calculate this area with previous versions of FSET, just noticed the issue with latest version on this particular area. Maybe it was one case only, i have done a few other, smaller areas with 208 and did not notice slow down - will monitor further.

[stackTom]

Sounds good. Some areas are just inherently slower as the coasts are more complicated and have more water bodies.

[Artpom]

Now i understand my 'slow calculation' observation. Yesterday i started another bigger 'project' - Italy ( 675 tiles ). Again noticed significant slow down - every resample calculations took an hour to finish, even initially scheduled for 28 minutes. Slow down starts after reaching 80% of resample calculation progress, last 20% is done with 'iceberg movement speed'. The thing is that slow down lasts as long as textures are downloaded - processor is too loaded with all calculations. Now after few hours when there is no 'additional process' going on, and only 4 instances of resample calculations going on, there is no visible slow down, and each process takes around 30 minutes. Due to that another idea for FSET feature came to my mind - to have option to postpone resample calculations until download and preparation is done for all tiles.

[stackTom]

That is definitely a reasonable request. However, I wonder if it'll result in any speed up? After all, the work has to get done regardless. Not sure if the order of the work matters? I will give it more thought and try to test and see if it's worth doing that.