jdolan / quetoo

Quetoo ("Q2") is a free first person shooter based on id Tech2. GPL v2 license.
http://quetoo.org
204 stars 28 forks source link

Pan's buglist of map editing issues and other errors #658

Open Panjoo opened 2 years ago

Panjoo commented 2 years ago

I will report here the issues that I come across while finalizing the DM maps. The more I actively work in the mapeditor again and test all kinds of things ingame, the more little bugs I find. (If I encounter more things I'll add to it here.)


Buglist:

[new]

Paril commented 2 years ago

Yeah, his fix will be merged in soon. Probably today.

jdolan-chwy commented 2 years ago

Yea, sorry Pan. In my update last night (zzzz) I forgot to mention that I haven't merged these changes in yet. They are just sitting on my hard drive while I continue to work on them. Sorry.

Panjoo commented 2 years ago

No problem.

In case you think I'm imagining things (always a possibility) now check this out. Upon compiling fragpipe again with a different and lower ambient, the light compile just seems to repeat the Indirect lightmaps stage... The 3rd time it starts with Indirect lightmaps it takes for ever and I have to cancel it.

double-lightmaps

jdolan-chwy commented 2 years ago

Oh what the F!? What does your command line look like? OH! You are running --bounce aren't you? Yea.... that will absolutely destroy your compile times 🤣 You're effectively building up stronger and stronger lights, and pointing them at each other 🤣. That code path needs some work...

Panjoo commented 2 years ago

quemap -bsp -light --bounce 3 maps/fragpipe-new.map

(!) but it also happened without adding --bounce.

Edit: Well, I knew --bounce affects compile times, but I've compiled so many times with and without it, and the way it repeats itself is not always the case.

jdolan-chwy commented 2 years ago

I don't think that's true. I'm quite certain that you'll only see multiple passes of Indirect lightmaps.. with --bounce. I have never seen it (I never use --bounce), and I have baked our maps hundreds of times in the last couple of months.

Panjoo commented 2 years ago

If you dont use --bounce you won't see it, but I mean that I've also compiled hundreds of times with bounce, and it never seemed to take so long on the 2nd and 3rd pass as with fragpipe before.

Edit: you're right about it repeating the indirect lightmaps stage due to that bounce setting. But as I said, sometimes it just seems to completely hang and other times it's "fast".

Now this is tested with and without --bounce, which obviously doesn't work and it apparently also caused the uneven lava lighting? I think this is a bit of a problem, I know it's under development but I kind of expect these lighting tools to work when I try to use them. I want my money back!

With bounce 2 - LOL: with-bounce2

Without bounce: without-bounce

jdolan commented 2 years ago

I might kill off --bounce. I'm not sure it really makes sense for us, and it's clearly quite broken.. Would this bother you? Assuming we have a properly working single-pass indirect lighting run, of course..

jdolan commented 2 years ago

Okay, here goes:

quetoo015 quetoo016 quetoo017 quetoo018 quetoo019 quetoo020 quetoo021

These are all using the unmodified fragpipe.map that is in Git. No changes to the map or materials have been made at all. Things that are fixed:

_Background: since we added the _size code over a year ago, every light source is actually a collection of points, not just a single point. I refer to these as "sample points." This enables a lot of nice looking effects, but complicates the code._

  1. Light bleed. This was due to the old light _size code that generated sample points around the light origin. This code checked each point for solid / not solid, but did not run a trace from the light origin to the sample point. So if a sample point got projected through a wall into a valid position, it would still be used. This is no longer the case. I trace from the light origin to each light sample point to ensure it is valid.
  2. Indirect lights no longer light their neighbors. There was already a check to not self-light. But because patch lights project their sample points out a few units from the wall, they were actually lighting their neighbors. This was the cause of the "Why is the red light climbing all the way up this wall?" and also "Why is this large floor peppered in hot spots?"
  3. Patch size deficiencies. The old code calculated attenuation from the light origin, not from the light sample point in question. This resulted in hot spots and cool spots near large face lights, and required you to dial patch_size down to get nice looking face lights. The new code now correctly calculates attenuation based on the actual light sample point being checked.
  4. Sunlight now diffuses correctly, and does not require a massive _size to do it. The old code pushed the sun origin MAX_WORLD_DIST away, so even a _size of 512 units wasn't going to produce much variation. Sun _size now kicks in at a distance of 1024 units, and so even the default sun _size of 32 units produces nice soft edges. You can crank this up to get nice looking penumbrae.

Optimizations:

  1. For point lights, spot lights, patch lights and indirect lights: rather than calculating sample points over and over for every single luxel, I calculate these up front when the light is created. For sun lights, I calculate a bouquet of directional vectors.
  2. When lighting a luxel, the light sample points are first sorted by distance, and tried in order. The first sample point with a successful trace is used, giving the best possible attenuation for that luxel. If we reach a sample point for which the luxel is out of range, we can break early.
  3. For patch and indirect lights, use the face plane to perform a sidedness check against the luxel origin. This allows us to skip a ton of traces.
  4. Kind of covered in fixes above, but, by not lighting coplanar neighbors, indirect lighting got a nice boost.
  5. Don't bother antialiasing indirect lighting. It's just not necessary.

The bake you're looking at is a full compile (quemap -light --antialias maps/fragpipe.map) and took 129s on a 5 year old laptop.

jdolan commented 2 years ago

These fixes and a few more are now in develop @Panjoo, so you can update your snapshot. A couple notes:

  1. You've probably already figured this out, but when I added CONTENT_ATMOSPHERIC for fog, so that fog would not eat mist (decals), some maps had fog volumes on them that were explicitly set to CONTENTS_MIST. Hallways, Slimy Place, Rage and Lava Tomb all have this issue. Those fog volumes should just be cleared of all contents so that quemap can auto-set them to atmospheric based on the texture name common/fog.
  2. You may find that radiosity levels need adjusting (likely down). This is because, despite my efforts to not change the actual lighting model while addressing these bugs, some changes were a consequence of fixing these bugs. I would recommend trying different -light --radiosity .666 type values until you find what looks good, and then you can just tuck that radiosity value into worldspawn.
  3. With the latest batch of changes, the default patch_size is now 128. This speeds up direct lighting (if you're using face lights) and dramatically speeds up indirect lighting. I'm able to compile Edge in 20 seconds. Hopefully you see a nice performance bump here, and recompiling these maps to tune lighting isn't as tedious as before.
jdolan commented 2 years ago

Also, @Panjoo, I just became aware of an issue compiling Lost Hallways, Gehenna, Slimy Place, etc.. a lot of these were crashing due to some of the contents and visibility changes made a couple weeks ago. With @Paril 's help, I got those fixed as well. So.. apologies if those were giving you a hard time. I didn't realize there was a regression until this morning. But you should be able to cleanly compile all of our map files with the latest snapshot.

jdolan commented 2 years ago

Oh! One more thing. I fixed face lights on inline models. Lights on brush models will not light the world (or the light grid), so you can put that little light back on the func_train if you want to. It will only light the train itself.

However, the func_waterin the Frag Pipe is an example of where we were benefiting inline models lighting the world. You'll notice that, now, the inside of the pipe is not glowing red as it was before. You can just add some red point lights inside the pipe to achieve that effect again.

Lmk if this works for you, or if you feel it should be reverted. Either way, I think light-emitting-brush-models are kind of a rare thing. I can't think of another map besides Frag Pipe that has one, nevermind two!

Panjoo commented 2 years ago

Excellent, I'm glad you guys are able to fix and improve these things. Thanks for the info and tips, will save it all in a document with the rest. I will be experimenting with lighting (values) until things feel just right again. I'm sure I can get these maps looking better, including some texture upgrades. Some already do but no spoilers yet.

(Note that in that old fragpipe it still has global fog, which makes things look brighter and more colored than it really is. For some reason the newer global fog looks different than the old global fog, which seemed to adjust to distance. Can't really put my finger on it, but it doesn't look like it's adjusting, the thickness only decreases when you move closer to solid objects. You really need super low values and even then it just looks like a hazy layer on every surface. I don't think I'll be using it in many maps, at least not in the smaller ones, the local fog volumes on the other hand look great and are a real asset.)

One last question for now. In the entdef it says: radiosity : Global indirect light scale, a single positive scalar value (e.g. 1.675) But what does it do to the lighting, in lay terms?

Do you mind if at some point I clean this thread up a bit, and hide some of the screenshots?

jdolan commented 2 years ago

Yea we should probably reimplement basic global fog instead of the global volumetric stuff. It would look and perform better. I’ll open a ticket. It shouldn’t be much work.

Radiosity, in layman’s terms, is one method for scattering light to make it look more natural. What our implementation does is turn every single face in the map that received direct light, into an indirect light source. It’s literally the equivalent of setting SURF_LIGHT on every single face, but with a small value. The light color is a 50/50 mix of the incoming light color and the face material color. So a green face receiving white light will emit green light. A blue face receiving red light will emit purple, etc. The radius of the indirect light is based on the intensity of the incoming direct light. The global radiosity scalar is just a multiplier for indirect light radius. That’s how it works. Pretty simple really.

The default radiosity scalar is 1.0, which means that the light received is reflected in equal intensity. That’s not very realistic, and looks artificially bright in my opinion. So that’s why I suggested the 0.666 as a good starting point for some subtle but pleasant and natural looking indirect light.

Paril commented 2 years ago

The main benefit to the global volumetric one is being able to take advantage of noise. Tokays for instance looks really good with the hazy fog spots. Unfortunately it also doesn't look great at distance since it only samples so far ahead of your view, so it's better for short range fogging.

Both can co-exist, which I think is the better solution.

jdolan commented 2 years ago

@Panjoo Are you tackling more than just q2dm* right now? Are you updating Grunt, OMF, Aerowalk, Chastity, etc as well? Just looking to coordinate and not duplicate effort.

Panjoo commented 2 years ago

Yes, basicly I'm switching between multiple maps back and forth, comparing things and making sure the materials of shared textures are using the same surfacelight numbers. I'm making good progress but I'm also trying to get some sort of uniformity going in the maps in terms of light values. Like for instance setting brightness 2 in the worldspawn for all maps. Also in a couple maps I found textures that really needed attention or even complete redoing like this, and some of the skyboxes and water textures needed work. Usually it is this texture finetuning that is the hold-up, but this is almost done and I can begin packing some pk3's.

jdolan commented 2 years ago

Would you find it helpful if we had the ability to provide common texture material files and then include them by name in map-specific material files? In other words, have e1u2.mat and then edge.mat has an

#include “e1u2.mat”

At the top? Just something I have considered over the years but never built. It would allow us to reuse common texture sets more easily (q4power, Rygel, ..). We could bundle the shared mat file up with the textures themselves in textures-q4power.pk3.

We don’t have to do any of this. It’s just something that has crossed my mind.

Panjoo commented 2 years ago

It could be helpful to have common texture material files as long as the sets have good _normals and are tuned right.

I personally prefer to be able to tune anything if desired, instead of working with presets. Not sure if that would still be possible when it's bundled with a pk3. But if the map's own .mat file is going to override anything from a shared .mat file, then I'd say do it.

jdolan commented 2 years ago

@Panjoo Just a heads-up that I pushed a change in our fog shader that attempts to address just how volatile our fog volumes are. The way the shader works now, is it scans a the lightgrid for fog every 32 units, and takes the most dense sample that it finds as that pixel's fog value. Previously, the shader added up all samples for each pixel and averaged them. The problem with that approach is that fog will almost disappear at a distance, and will be nearly opaque up close. I'm sure you noticed this.

Now that we have old-school global fog, our volumes are mostly for smaller areas, and are meant to limit visibility or add ambience. I think this technique accomplishes that much better. However, you may find that existing fog volumes need tweaking to look good. Dropping the density of the fog volume is probably all that is required.

Here's the difference. Before, with averaged samples: image

And after, with max sample: image

Clearly, the 2nd shot shows the fog more visibly. But what you can't really see from a static screenshot is that the fog looks exactly like that at all angles and distances, inside the volume as well as outside of it. It's a very nice effect, imo, and suits are needs better.

Panjoo commented 2 years ago

...fog will almost disappear at a distance, and will be nearly opaque up close. I'm sure you noticed this.

I did, it just looked unnatural before, that's why I rather had old-school global fog back.

...what you can't really see from a static screenshot is that the fog looks exactly like that at all angles and distances, inside the volume as well as outside of it.

It sounds like a very good change. I haven't had the chance to actually test and study this yet, but I will. I think I've mentioned this before but these local fog volumes are a real asset in a mapdesigner's toolkit.

Panjoo commented 2 years ago

Found something else but I don't know if this is from this change. I noticed that my light_sun values need adjusting again, the _size and light values appear to have been affected. Overall my sun light spots look somewhat dimmer and softer than before, so something definitely changed.

jdolan commented 2 years ago

Yes sunlight was basically not attenuated correctly in some of the recent snapshots. It was basically all or nothing. You may have noticed that even small sunlight values like 10 produced full bright results. So I fixed that I think Tuesday or Thursday. Sorry!

Panjoo commented 2 years ago

Okay, next problem...

In Edge I'm trying to use my fern model that I made as mapobject, but after some of the changes the lighting on this mapobject is different now and not for the better. Looks kind of fake and out of place now. Almost as if the bottom sides of the leaves are not receiving light. Not even if I shove a light-entity underneath, and not with a higher worldspawn radiosity setting. See the difference in these old and new screens:

older light compile: miscmodel-light recent light compile: miscmodel-light2

jdolan commented 2 years ago

Well, I did fix sunlight being full-bright the other day, so it would make sense that these are darker now. But I agree that the 2nd shot looks a little too dark. The leaves seem darker than the ground behind them.

Can you r_draw_bsp_lightgrid 1 and attach those screenshots? I wonder if the sun light direction is to blame.

Panjoo commented 2 years ago

I've tested other sun angles, without much improvement. But here is that previous testmap again; miscmodel.pk3. This still has _size 384 in worldspawn, so you might want to change the settings.

jdolan commented 2 years ago

quetoo008

Hm. I dunno. This actually looks 👌 to me. I deleted the _size key from the sun and baked the map locally to check.

Panjoo commented 2 years ago

Suddenly I see what is up... I am still compiling with-light --no-indirect - mainly because of the lava light errors that I was fighting in fragpipe. (My bad, this is probably not a bug so I'll delete this from the comments later on to keep it clean.)

Although, compiling with default indirect light does have strange behaviour in some spots as seen in this image from a newer version of fragpipe, that only uses one sun with a straight down angle;

indirect-sunblock

jdolan commented 2 years ago

Makes perfect sense. The sunlight is being reflected off of the tops of those beams. It looks a bit unnatural just because it’s too bright. This is why I suggested a radiosity value of ~.666 as a reasonable starting point.

SpineyPete commented 2 years ago

We do have angular attenuation (spotlight falloff) on patches, right?

jdolan commented 2 years ago

We do not, actually. Think we should?

SpineyPete commented 2 years ago

Yeah, spotlight style dot product where the surface normal is the direction. That's probably going to change the look of the maps a bit. Can't imagine it being worse though.

SpineyPete commented 2 years ago

Fwiw iirc it was set up that way on Void's prototype.

Panjoo commented 2 years ago

This is why I suggested a radiosity value of ~.666 as a reasonable starting point.

That screenshot with the beams is from a compile with a lowered radiosity of .07 as suggested. Perfect sense or not, at first glance it looks like a lighting error to me. I didn't realize the radiosity effect would still be this strong at 0.7

But... I've now set it to 0.3 ( 0.4 also works) and it looks a lot more natural, and it also removed some of the unwanted red spots as seen in this screenshot below. The red light is perfectly horizontal and above the walkway, and I didn't expect it having to do with the radiosity, I mean the lava light is coming from below, why would it be reflected there?

radiosity-07

jdolan commented 2 years ago

quetoo010

I added spotlight-style angular attenuation to indirect patch lights, per @SpineyPete 's suggestion, and those artifacts are gone. This is with radiosity 1. I added the code path for angular attenuation for direct patch lights as well, but did not enable it, as I think it changes the lighting model too much and would frustrate you 😁

Panjoo commented 2 years ago

Nice one, another improvement.

I added the code path for angular attenuation for direct patch lights as well, but did not enable it,

Ehm, by direct you mean no-indirect? It's starting to confuse me.

-light = indirect lighting (the default) -light --no-indirect = direct lighting

Other than shortening compile times and bypassing these sort of artifacts, would there be any reason for a mapper to use --no-indirect?

Paril commented 2 years ago

It'd be used for "fast" compiles - sort of like how you'd omit -bounce in Q1/Q2 tools.

He means "direct patch lights" as in non-bounced surface lights. Any direct light comes directly from the source; indirect comes from bounces.

jdolan commented 2 years ago

That's true, skipping indirect lighting has historically been a "fast bake" option. But now, --no-indirect is really meant for debugging purposes. Our compile times are so damn fast, that I'm hard-pressed to recommend --no-indirect as a "fast bake." Maps look so much better with it indirect lighting, and the indirect lighting pass always takes less time than the direct lighting pass anyway (unless you crank radiosity up to 2 or something).

Basically, at this point, I use --no-indirect to see which pass is creating problems 😁

But to @Paril 's point, he's correct. We have two kinds of patch lights:

  1. Direct patch lights (SURF_LIGHT) set by the mapper via surface inspector or materials.
  2. Indirect patch lights; every face that received light in the direct pass becomes a SURF_LIGHT for the indirect pass.

I enabled spotlight attenuation on indirect patch lights only. The code is also there to turn it on for direct patch lights, but it makes the maps a good bit darker, because we are mostly using patch lights, and with this change, they cast less light! So I think it's best to leave direct patch light behavior unchanged.

Panjoo commented 2 years ago

Thnx, it's all clear to me now.

jdolan commented 2 years ago

Btw @Panjoo, I'm not sure if you follow the commit logs, but yesterday I added a debugging tool you might find helpful: r_draw_bsp_lightmap. Setting it to 1 will show you diffuse + ambient + radiosity, setting it to 2 will show you light direction. On mesh models, it will show you the same values, but from the lightgrid.

If you use it in conjunction with r_texture_mode GL_NEAREST you can see exactly how each luxel was lit.

jdolan commented 2 years ago

@Panjoo FYI, I'm going to add in "auto-Phong" functionality in quemap that will automatically apply smooth shading with a per-material tolerance value. This will eliminate the need for painstakingly setting SURF_PHONG on most rounded objects. The threshold will be a simple scalar (0.0-1.0) and configurable per material, so if you want to disable it, or force it to always on, you can do that. The actual brush side surface flags will override, so you can disable auto-Phong in the .mat file but still enable it per-face if that's desired.

Basically, I was looking at Jester's gothic map and thinking.. there's no way I'm going to correctly Phong this in Radiant 😝

quetoo011

Let me know if you have any concerns about this feature. I should be able to knock it out in the next evening or two.

Panjoo commented 2 years ago

At this point I don't know about extra features when maps are almost done. But as long as it can be disabled I'm fine with anything that can help simplify the process. Right now I have more concerns about the playabilty the game, again, ie. the fps performance. (see the buglist at the top).

jdolan commented 2 years ago

Flares are drawn without the depth test. This works mostly well on solid, opaque geometry. But clearly has issues when the player can occupy opaque geometry. We'd see the same issue with CONTENTS_MIST. We should probably re-enable the depth test for flares. I don't think this is a big deal and will get to it soon.

The sprite effects on water are expensive, and they could still use some optimization for sure. They are white nuclear blobs because there are too many ripples being spawned there. And that's also why they are tanking your framerate. No need to panic 😁

cl_max_fps 0 (the default) matches the soft framerate cap with your vertical refresh rate. This is the most desirable configuration for most systems. It provides a soft cap that reduces tearing, without incurring the input latency penalty of vsync. You can set cl_max_fps -1 to disable all capping and see what your max framerate actually is.

Your other FPS observations are interesting. Do any of the following dramatically affect your performance?

r_fog_density 0 r_bloom 0 r_draw_material_stages 0 r_blend_depth_sorting 0

Do any of these move the needle for you?

jdolan commented 2 years ago

Side note, I think translucent water with fog and material stages looks so much nicer than opaque water. You end up with some really nice light interactions between the lightmapped water surface and light-absorbing volumetric fog. Just my $0.02.

Also, my new favorite trick is adding misc_dust underwater to emit bubbles rising from the deep. I'm doing this on my Cistern remake, but it would look good on Edge, Tokays, Pits, and Hallways, too..

// entity 58
{
"lighting" ".666"
"_end_color" "0 0 1 0"
"_color" "0 0 1 1"
"density" "6"
"_size" ".5"
"acceleration" "0 0 -.2"
"lifetime" "3.2"
"velocity" "0 0 28"
"sprite" "bubble"
"classname" "misc_dust"
// .. brushes
}

You can tweak acceleration and lifetime so that the bubbles expire as they reach the surface.

Panjoo commented 2 years ago

I agree it may look nicer but on certain maps it negatively affects the gameplay. I know this because I still play the Q2 versions of these maps on a regular basis and I want to deliver truthful remakes. If all water is translucent it's going to take away the element of surprise. Combined with the too slow player movement in water (it barely lets you move side to side) I don't think it's a good choice to make it so that you won't be able to hide in the water or pop out of it by surprise. Especially in Edge, Pits and Hallways. Perhaps thick fog might work, but then it should not be too expensive.

If normal water without any extra fx is already making things run slow I don't think I want to play around with more fx stuff like fog and bubbles at this point.

Your other FPS observations are interesting. Do any of the following dramatically affect your performance? r_fog_density 0 / r_bloom 0 / r_draw_material_stages 0 / r_blend_depth_sorting 0

Hmm, none of them seem to make a noticeble difference. It's too late now but I'll do some more testing tomorrow.

Been getting some other errors that I've never seen before, probably due to mangled brushes; max_draw

R_Debug_Callback: OpenGL (API; Medium) Performance [id 131218]: Program/shader state performance warning:
Vertex shader in program 11 is being recompiled based on GL state. 
 source: > R_DrawSprites (r_sprite.c:501)
jdolan commented 2 years ago

Fog is a constant cost. Whether there are 0 volumes on a map, or 1000, the same exact calculations are performed. You can see what impact fog has by setting r_fog_density 0 to entirely disable the feature, or by playing with r_fog_samples, where higher values will slightly improve quality but reduce frame rate. But again, individual fog volumes do not affect performance at all. Material stages are more efficient than they were in all previous iterations of the engine, too, so I wouldn’t shy away from those.

I’ll spend some time digging into performance issues this weekend. What resolution do you run at, btw, and are you on a 4K display?

SpineyPete commented 2 years ago

Yeah the water ripples are a bit of an FPS killer, I'll have to redo them and see if I can reduce the overdraw and limit the rate at which they spawn. Like MG for instance probably spawns too much and too large ones, LG is even worse. Just limiting the spawn rate and size might be sufficient, alternatively I could just try to make the effect look different and take less surface area (always open to suggestions / inspiration from other games). The hyperblaster impacts were another one that were a bit of a hog before, but for me at least this seems to have been resolved.

Panjoo commented 2 years ago

Yes, for a while now I've been using _bind "f" "toggle r_fogdensity" to check how the density looks compared to no fog at all. Same with _bind "c" "toggle rcaustics"

As far as I can tell it's not the fog that has the most impact, but it's translucent liquid volumes and things like (blended) window textures that are eating a big chunk out of my overall fps. I don't have all that much to spare so yeah, every significant drop makes me frown. I'm just surprised that even one body of translucent water has this much impact.

What resolution do you run at, btw, and are you on a 4K display?

Hah! Wouldn't that be enormously newbish? Reporting about bad fps in "great panic" while using a 4K display on a PC with 2GB vidcard that's already struggling to get a constant 200 fps in boxmaps in Quetoo? I'm running at 1680x1050 resolution on a 22", but I was already thinking about switching to 1440x900, that should make some difference I'm sure, but still.

Is there a way to add borders? That's how I usually like to play anyway, since I prefer smaller views for fast games. _cl_view_sizedown and _cl_view_sizeup stopped working long ago.

Panjoo commented 2 years ago

... see if I can reduce the overdraw and limit the rate at which they spawn. Like MG for instance probably spawns too much and too large ones, LG is even worse. Just limiting the spawn rate and size might be sufficient,

The textures are quite good actually, there's no need to edit them I think. But I doubt we need so many images in sequence to make a convincing splash. I'm counting 75 splash images... I'm thinking a combination of limiting the spawn rate and less 'animations' will be sufficient.

SpineyPete commented 2 years ago

It's good you reminded me about the amount of frames. Actually those textures were originally made for a railgun impact, which had a much longer animation and I tried to keep all animations around 20-30 fps to make them look smooth, but that's obviously too much for this one, so I can probably cut those considerably.