Add future update to fully define settings and perms interactions.

[CleanerOrc]

Could you guys add an improvement to your To Do list to update the configuration / perms info to give a reasonable explanation for all of the config options and permission interactions..

For example the problem I encountered using the default chunk-wait value 0 causing paste operations to duplicate small sections of the copied area.

There's no mention anywhere that the chunk-wait setting could have this effect. Would be great to note this symptom and what to do about it in the description of chunk-wait.

[boy0001]

Updated documentation: https://github.com/boy0001/FastAsyncWorldedit/wiki/Configuration

Would you add any other info there?

[CleanerOrc]

For chunk-wait-ms:

--------------------------------
# The time in milliseconds to wait for a chunk to load for command preprocessing (50ms = 1 server tick, 0 = Fastest)
# Actions which require loaded chunks (e.g. copy) which do not load in time will use the last chunk as filler.
chunk-wait-ms: 100
--------------------------------

But It just would not register that "using the last chunk as filler" would show up as bands of duplicated blocks in my copy.  Especially since most of the whole copy was the same band of chunks filling the whole rest of the copy.   Makes sense now after you mentioned it and I thought about it though.

Anyway, I''d tweak the desc to be really clear by including a symptom of it being too low, like:

---------------------------
# The time in milliseconds to wait for a chunk to load for command preprocessing 
# (50ms = 1 server tick, 0 = Fastest).
# The default value of 100 should be safe for most huge (200+ Million block) operations.
# Actions which require loaded chunks (e.g. copy) which do not load in time 
# will use the last chunk as filler, which could cause bands of duplicated blocks
# in the copy.  If that happens, use a larger value to give more time for chunks to load.
# Decreasing will speed up chunk based operations, increasing will slow them.
chunk-wait-ms: 100
-------------------------------

I hate to mention it, but I''m guessing there''s no other way except waiting a fixed period, to determine
that chunks have loaded properly?

Far as the rest of the settings, Give me some time to think about the other config options and I''ll comment questions and suggestions.  In general, seems if you explain what happens when values are raised or lowered and give experiential examples - helps users tweak settings.  

For example:
# Max number of block changes
    max-changes: 5000000
# Number of times WorldEdit can check the block at a location
    max-checks: 5000000

Does max-changes mean the number of blocks replaced by copy?  Or only when //replace is used?
When does WoldEdit check a block at a location?  What operations does this affect.

I want users to have WE in their regions, but I''m not a WE expert that understands the inner workings of WE.  Need to know what in WE these settings affect.  Unless you intend users not to tweak these unless you recommend changes...

[boy0001]

I hate to mention it, but I''m guessing there''s no other way except waiting a fixed period, to determine that chunks have loaded properly?

It usually waits about 25-50ms for the chunk to load, more if the server is lagging. Having a value of 100ms does not force it to wait 100ms if the chunk loads in 10ms. A cutoff point is required as if a chunk is never going to load (for whatever odd reason), there still needs to be a way to continue the operation.

A value of 0 is faster simply because it doesn't bother loading the chunks / waiting.

Does max-changes mean the number of blocks replaced by copy? Or only when //replace is used? When does WoldEdit check a block at a location? What operations does this affect.

max-changes is number of blocks placed into the world.

max-checks is the number of times WorldEdit checks a block (not including history) Whatever commands these correspond to: count distribution green recurseHollow simulateSnow thaw getHighestTerrainBlock recurseHollow copy makeForest setBlockIfAir regenerate any time you modify a block with NBT probably some other stuff

A good value might be setting it the same as max-blocks. It's just there to limit commands which don't modify any blocks e.g. //count

[CleanerOrc]

@boy001,

Say, What’s your goal for the config file? Do you want users to understand how all of the options work, OR, maybe just a few, but the rest are there in case you need to have them change some value to resolve a support ticket? ( Anyway, you mentioned still being in Beta and waiting to see how stable the algorithms were. Maybe things are changing fast and you just haven’t gotten to beefing up the docs yet. )

But IMHO, When programmers write docs, they write from a programmer’s view of someone who knows intimately and takes for granted how the code works. Users generally don’t have a clue how the code works, so if you want them to make informed changes, you have to explain how changes affect internal behavior and give some symptoms of going too far. At least for the settings you want them to make on their own without bothering you.

I can donate some time to comment on the options you want us to understand, like I did for chunk–wait–ms. I can’t donate a lot of $$ at this time, but I’d be happy to help strengthen the docs. I think this is an extremely useful plugin!

I’d have to ask questions and get feedback from you for each option that I can’t understand. If that would be helpful, I’d need to know how you want to communicate – email, or thru maybe this issue that stays open till we’re done?

Anyway, A great example of giving enough info are the changes you made to the latest chunk–wait–ms desc along with the additional comments to me below. I suggest you also add the following info to the desc! In my mind, this really well describes how your chunk handling algorithm works. I wouldn’t have asked for support if I’d seen this in the config description.

Also, as this is (to me) a crisis situation, Is it possible to detect when a chunk load failure occurred? Is there a way to issue an error msg that this happened? Maybe only the first time, or – count the number of times and let them know how many times after? Warning the user it happened RATHER than make him have to examine the copy to see if it worked OK (ARGH) would increase trust. An even more professional option would be to include a suggestion in the error msg to increase the chunk–wait–ms setting or reduce the selection size. At this point I don’t trust that it won’t happen. Puts me on edge as I often do large copies as it’s a little easier than using MCEDIT…

Please consider adding to desc:

It usually waits about 25-50ms for the chunk to load, more if the server is lagging. Having a value of 100ms does not force it to wait 100ms if the chunk loads in 10ms. A cutoff point is required as if a chunk is never going to load (for whatever odd reason), there still needs to be a way to continue the operation.

A value of 0 is faster simply because it doesn't bother loading the chunks / waiting.

[boy0001]

IRC: http://webchat.esper.net/?nick=&channels=IntellectualCrafters Skype: jesse_paleg Or on here is fine.

Also, the Wiki should allow anyone to contribute. I get notified when changes are made, so I can check any info you add.

[CleanerOrc]

OK, thanks. Appreciate being able to contribute.

From: Jesse Boyd [mailto:notifications@github.com] Sent: Wednesday, April 20, 2016 1:57 PM To: boy0001/FastAsyncWorldedit Cc: CleanerOrc; Author Subject: Re: [boy0001/FastAsyncWorldedit] Add future update to fully define settings and perms interactions. (#37)

IRC: http://webchat.esper.net/?nick=&channels=IntellectualCrafters Skype: jesse_paleg Or on here is fine.

Also, the Wiki should allow anyone to contribute. I get notified when changes are made, so I can check any info you add.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHubhttps://github.com/boy0001/FastAsyncWorldedit/issues/37#issuecomment-212535555

[CleanerOrc]

Here are the remaining issues I’d like to beef up desc for. NO RUSH. If you could give a good explanation, would help not having to ask follow–up questions. :–) Figured I’d just send a whole list at once – you could pick at it as you have time.

CONFIG:

max-memory-percent: Verify that this does not affect the following users: OP’d users, those with fawe.bypass, those with fawe.admin and who have used the /wea command.

require-selection-in-mask: I’m Not really familiar with WE masks. Can you give a little more explanation to what is required if this is true?

Queue Settings: extra-time-ms: 0 ASK: Is there any more info you can give to explain to a newbie when this should be changed, and when it’s been changed too far? Symptoms?

max-wait-ms: 1000 target-size: 64 ASK: You mention the preprocessing stage and that it’s more efficient for edits to complete it. Can you give any more info for a newbie on how to know when to change these values and symptoms for when they’ve been changed too far?

PERMISSIONS:

User permissions: TODO ask about [fawe.limit.] From Config: [# To grant someone a higher limit, give them fawe.limit.] It’s not clear what can be. Typically when you use “group” in permission nodes you think user group like admin, owner, etc. But the comment in the config section makes me think you can add a permission node to spec a config option value like Fawe.limit. max-changes.2000000 Can you verify? If so, mind if I change “” to “” or “” throughout? Assuming only the settings under Limit: tag may be set this way.

Tks – co

From: Jesse Boyd [mailto:notifications@github.com] Sent: Wednesday, April 20, 2016 1:57 PM To: boy0001/FastAsyncWorldedit Cc: CleanerOrc; Author Subject: Re: [boy0001/FastAsyncWorldedit] Add future update to fully define settings and perms interactions. (#37)

IRC: http://webchat.esper.net/?nick=&channels=IntellectualCrafters Skype: jesse_paleg Or on here is fine.

Also, the Wiki should allow anyone to contribute. I get notified when changes are made, so I can check any info you add.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHubhttps://github.com/boy0001/FastAsyncWorldedit/issues/37#issuecomment-212535555

[boy0001]

max-memory-percent:

Effect anyone who doesn't have bypass enabled /wea or fawe.bypass or fastmode //fast.

require-selection-in-mask:

It's an old setting, probably going to be removed at some point. The mask refers to the region restriction, e.g. A worldguard region you are added to. If you have pos1 or pos2 outside this region (and you don't have bypass enabled) it will cancel the command and tell you.

extra-time-ms: 0

Never really needed to change it myself, I just added the option for those who like to tweak everything. If you want to halt the server during an edit, you can increase the number to something crazy high. Decreasing it (negative) to reduce load is an option, but ideally you shouldn't ever need to do that.

fawe.limit.<group>

So in the config you will see a section called "default" with a set of limits. The group here is default, so the permission would be fawe.limit.default (though since it's default that's probably redundant. Copy that section to create different groups with any name you want. If you think group is confusing you can change that, but it is a group in the sense of the word.

e.g. To add an admin group (fawe.limit.admin)

limits:
  admin:
    max-changes: 50000
    max-checks: 50000
    max-iterations: 50000
    max-entities: 1337
    max-fails: 0
    max-blockstates: 1337
  default:
    max-changes: 10000
    max-checks: 10000
    max-iterations: 10000
    max-entities: 1337
    max-fails: 0
    max-blockstates: 1337

[CleanerOrc]

On /wrg. Does it select any region you are in (based on the fawe.), OR, only regions/plots that you are in AND you own or have specific access to.

What does it do if you don’t have config region–restrictions set true?

[boy0001]

Missed some:

max-wait-ms: 1000

Already covered max-wait-ms. Increase it when chunks fail to load in time. I'd say going over a second would start getting a bit ridiculous as it shouldn't take that long to load a chunk.

target-size: 64

A bit of background on how FAWE works. All commands are executed on another thread i.e. it runs alongside whatever the server is doing. A local queue is created for an edit in which all the blocks to be changed are grouped by chunk. The global queue (which the target size refers to) dispatches these queued blocks and makes changes to the world on the main thread. When command execution is complete, the local queue for it tells the global queue it is complete. The global queue prioritizes by insertion order, so older local queues have their changes made first. Local queues that are complete get priority over ones where the command is still processing, and processing local queues are only dispatched when the global queue has > target-size chunks.

Some commands such as //deform and //smooth may not deform or smooth properly if they are dispatched while they are still processing. (these usually don't modify anywhere near 64 chunks). In terms of efficiency, it's only really where the global queue may dispatch changes to the world, to then find out that the local queue modified that chunk and it has to do it again. For simple commands like //set with the cuboid selector it queues it by chunk already so that's not really a problem.

On /wrg. Does it select any region you are in (based on the fawe.<plugin>)

You have to be standing in it and have access to the region.

What does it do if you don’t have config region–restrictions set true?

It does absolutely nothing.

[CleanerOrc]

Ok, couple misc config setting questions. I’ll send a longer email on Queue processing that should wrap up my questions.

chunk–wait–ms: ASK: Why is chunk–wait–ms in the History: section of the config file? Is there a reason or just inertia?

VERIFY

There’s one local queue per user. Guess this is moot, as you can only process one command at a time…

Have you designed against multiple command threads per user as was available in AWE?

extra-time-ms: 0 ASK: Could you explain where the extra time is added or removed in the algorithm? In other words, specifically, what can you tell the user who wants to tweak about how this setting affects the performance or stability? Can you think of any Symptoms of extra–time–ms being too negative?

VERIFY statement: “Should not need to change this. Increasing it ( positive value) too high will probably cause the server to freeze, and decreasing it ( negative value) may reduce load on the server but should not be necessary.”

[CleanerOrc]

Here’s my perception of how the local and global queues interact from what I’ve seen so far. Can you help me either verify or correct my understanding of the algorithm?

Completed local queues are queued to the global queue when they complete, as a higher priority. But the global queue also queues lower priority incomplete local queue chunks in two cases:

First, It looks like target–size tries to speed up operations by risking to dispatch in–process local queues once any multi–chunk changes (ie, from //smooth, //deform, etc) should have finished – trying to reduce re–work that would occur if all the changes hadn’t completed and needed to be changed again, slowing down the operation.

Second, You mentioned that “FAWE is most efficient when edits have finished preprocessing, but larger edits may use too much memory to let them finish.” It appears that max–wait–ms tries to continue an operation when it’s likely that an incomplete local queue won’t finish because of not enough memory, indicated by the global queue being idle too long. In this case, it goes ahead and processes the incomplete local queue as is.

So,

1) If max–wait–ms is used to continue an operation that likely wouldn’t complete, is this a case where some expected block changes won’t happen (because of lack of memory), but the algorithm ignores this in order to complete the command?

2) If not true, could you help me understand how max–wait–ms really does affect the overall algorithm? Just not clear to me why you use it then. You also have a comment that “”.

3) Are both of these conditions required before incomplete local queues are processed (idle > max–wait–ms AND chunks in global > target–size), or are these conditions independent?

[boy0001]

Why is chunk–wait–ms in the History: section of the config file? Is there a reason or just inertia? When it was first added, only history (undo // redo) used the chunk-wait algorithm for fetching blocks. As I worked on the plugin more, different parts began to use it to fetch blocks in unloaded chunks.

There’s one local queue per user. Guess this is moot, as you can only process one command at a time… Have you designed against multiple command threads per user as was available in AWE?

It may be added at some later point. Given how fast the plugin is, there's not really much of a rush to add this.

Something that has been considered is having multiple edits affecting the same chunk. Because of how FAWE caches the state of the chunk, an edit will undo to an earlier state of the chunk (around chunk-wait-ms / 2) rather than the previous value. This isn't an issue for most things as things like a rollback never need precision to 50ms. The Global queue prioritizes by insertion order, so providing the target size is sufficient, there won't be issues of the blocks actually being changed at the same time.

Here’s my perception of how the local and global queues interact from what I’ve seen so far. Can you help me either verify or correct my understanding of the algorithm? So to clarify, the global queue technically has two queues. One queue for the local queues that are finished, and another for local queues that are still processing. It'll always go to the complete ones first.

If max–wait–ms is used to continue an operation that likely wouldn’t complete, is this a case where some expected block changes won’t happen (because of lack of memory), but the algorithm ignores this in order to complete the command?

Max wait isn't for that exactly. It's more there to make sure that the global queue isn't idling when it could be doing at least something useful.

The more important setting is the target size. Basically, let's say you do an edit of a billion blocks using fastmode. Without a target size, the queue would normally fill up faster than the max-wait-ms could empty it. When I say it wouldn't normally complete, that's only because the default WorldEdit behavior is to have everything loaded into memory before dispatching the changes. With FAWE and having the target size configured, we can ensure that (for most commands) every block will be placed and the server won't run out of memory.

The deform or smooth commands depend on the position of other blocks in the world and so the result will be somewhat different than if the processing completed before any changes are dispatched. Having a large enough target size will make this irrelevant.

Are both of these conditions required before incomplete local queues are processed (idle > max–wait–ms AND chunks in global > target–size), or are these conditions independent?

Either condition can be met for it to start processing incomplete queues. The complete queues must also be empty.

[CleanerOrc]

Thanks for all the time to explain this!!

Say, It’s looking to me like there should be a short section in the config page, above the actual config file, that describes the global and local queues and how they work in just enough detail so tweakers can get a handle on how the settings apply. The gist really should be described in one place rather in parts with the several config settings. This would include describing the priority of placing regarding blocks and the effects of multi–block commands.

I think it’d be much more readable than comments in the config code, especially since FAWE deletes the comments from the user’s config file anyway!

Would you be OK with that? I can write something up and send it to you or just take a stab at inserting in the config page.

Anyway, can you verify this updated understanding of how max–wait–ms and target–size affect the queue processing? (FYI, I’d like to include something like this in a section I mentioned above the config code, though I think it needs cleaned up some first. )

Question: User edits queue blocks, ordered by chunks, right?

So Is the unit being queued a chunk? Does the individual thing being queued move from local incomplete queue to the local completed queue when all the blocks for that chunk are queued, OR, are all the chunks from an edit placed on one local queue in incomplete state, and that whole queue is moved to completed state when the edit is finished?

At any rate, below, would it be accurate to speak in terms of chunks being in incomplete or complete state as opposed to queues in different states?

If so, the algorithm could be described as tracking chunk statea without having to explain that there are local and global queues and how they interact… Seems it would be more straightforward to understand?

Anyway, is the following accurate?

· Completed local queues are dispatched to place blocks as a priority before any incompleted queues. While there are completed queues ready to place, no incomplete queues are processed.

· If there are no completed edits ready to place, then any incomplete edits having more than target–size chunks ready to go are queued to place chunks. This tries to keep things moving and increase the placement speed performance. Dispatching incomplete edits is a bit risky though if the edit was from //smooth or //deform type commands since some of the blocks may need to be re–done. So the target–size setting attempts to lower this risk because it doesn’t usually take more than 64 chunks to process all of the multiple blocks that would need changed.

· Finally, If NO chunks have been placed in over max–wait–ms, then any incomplete local edits are also dispatched to place blocks. This tries to keep FAWE placing blocks instead of doing nothing – with the risk that some time might be wasted if the blocks have to be re–processed (e.g. for //deform or //smooth commands, etc commands).

With this algorithm, FAWE is able to really peed up block placement over using WorldEdit directly. And when FAWE is configured to save undo history copies on disk instead of memory, the number of blocks that can be operated on increases to almost unlimited without memory overflow.

[boy0001]

Would you be OK with that?

Sure

User edits queue blocks, ordered by chunks, right?

They are grouped into chunks. There is no required order in which the individual blocks are added to the queue.

So Is the unit being queued a chunk?

Yes. As a side note, the chunk can be modified up until the point it is dispatched by the global queue.

Does the individual thing being queued move from local incomplete queue to the local completed queue when all the blocks for that chunk are queued, OR, are all the chunks from an edit placed on one local queue in incomplete state, and that whole queue is moved to completed state when the edit is finished?

The local queue is a queue of chunks. The global queue is a queue of local queues. So changing to complete effects the entire local queue.

If there are no completed edits ready to place, then any incomplete edits having more than target–size chunks ready to go are queued to place chunks.

Incorrect. If the number of chunks in the global queue is greater than the target size it starts. i.e. it will work if there are multiple local queues which add up to more than the target chunks.

With this algorithm, FAWE is able to really peed up block placement over using WorldEdit directly. The speed improvements aren't specifically from the queue, it just enables some optimizations to be made elsewhere.

For example, default WorldEdit performs lighting, physics and sends packets for each block being changed. Because FAWE uses a queue that groups edits by chunk, more efficient algorithms can be used for the dispatching.