Mana Khemia 2/Ar Tonelico 2/Gustgames? Texture filtering problem.

[GoogleCodeExporter]

1) Did the game ever work correctly (i.e. not have this problem) on the
Official PCSX2 build or an earlier version of PCSX2 playground?
Older versions then GSdx-r1375 don't reproduce the problem. They have black
borders around the characters though. Software mode runs without any
texture problem, the crashes mentioned later on happen more frequently
though, and I notice a considerable fps hit.

2) What steps will reproduce the problem?
1. Texture filterting in any version earlier then GSdx-r1375 Hardware mode
both DirectX9 and 10

3) What exactly happens when you experience this issue (listing any console
errors or screen output you receive)?
Strange lines on characters during battles or cutscenes. Lines coincide
with unseen texture borders(?).

4) What version of PCSX2 are you using? On what operating system?
pcsx2-r1736
Windows Vista Ultimate x64

5) Please provide any additional information below.
Maybe relevant information:
Both games have occasional crashes. Nothing in particular seems to trigger
them. Doesn't seem relevant in relation to a GSdx problem, but I'm no
expert. :p

EE: Unrecognized COP0 op 40ff032f
(EE) Tlb Miss, addr=0x500021ff [store]
(EE) PC: 0x80000180     Cycle: 0x67af6920
--------------------------------------------
Core2Quad Q9300 @2.50GHz
4,00GBRam
Windows Vista Ultimate 64bit
NVIDIA GeForce 8800 GTX, Driver Version: 190.38

6) Attachments.
http://img29.imageshack.us/i/90984326.jpg/ Hardware mode(Mana Khemia 2)
http://img19.imageshack.us/i/64862411.jpg/ Earlier GSdx in hardware
mode(Mana Khemia 2)
http://img19.imageshack.us/i/24753268.jpg/ Hardware mode (Ar Tonelico 2)

Original issue reported on code.google.com by costa.am...@gmail.com on 24 Sep 2009 at 5:40

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Texture sampling / filtering issue again.
It's annoying, isn't it? :p

Original comment by ramapcsx2 on 24 Sep 2009 at 6:22

• Added labels: Plugin-Gsdx

[GoogleCodeExporter]

2) What steps will reproduce the problem?
1. Texture filterting in any version "earlier" then GSdx-r1375 Hardware mode
both DirectX9 and 10.

Excuse me, I meant "later", ;<

Also, someone was kind enough to provide me better screens with the same type of
issue in 2 other games:
http://img132.imageshack.us/i/gsdx20090924204807.jpg/
http://img441.imageshack.us/i/gsdx20090924205032.jpg/
http://img132.imageshack.us/i/gsdx20090924205549e.jpg/

Original comment by costa.am...@gmail.com on 24 Sep 2009 at 6:26

[GoogleCodeExporter]

This is pretty odd... playing the games and looking at the screenshots// it's 
the
sprite seams behind the front objects.  I don't see why there would be lines 
there as
the objects obstructing are completely opaque.

Original comment by necr...@gmail.com on 20 Nov 2009 at 5:24

[GoogleCodeExporter]

Reimplementing the 16-bit texture pixel shader's and simply fixing the black 
borders
directly in 1374 seemed to do the trick...
http://img205.imageshack.us/img205/3051/after2y.png

However the outside lines are jaggy, which is to be expected with a binary 
alpha. 
Gonna keep plucking at it.

Original comment by necr...@gmail.com on 28 Nov 2009 at 7:46

[GoogleCodeExporter]

So what's the status of the issue with the current latest GSdx?

Original comment by pcsx2gu...@gmail.com on 2 Dec 2009 at 10:49

• Changed state: Accepted

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Still an issue, black lines exist in filtering.  
I stopped working on this issue (actually working on a new graphic plugin...). 
Someone else pick it up please.

Original comment by necr...@gmail.com on 3 Dec 2009 at 12:30

[GoogleCodeExporter]

OK that's for the update. New GS plugin? Great, maybe you'd want to visit the 
dev
channel for PCSX2 (mail me for further info)

Original comment by pcsx2gu...@gmail.com on 8 Dec 2009 at 7:21

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I got an email notification on March 24 saying that there was a fix for this 
issue,
but I don't see it anywhere. :< Was the new version taken down to fix more 
bugs, or..
*confused*

Original comment by yuri...@gmail.com on 27 Mar 2010 at 6:52

[GoogleCodeExporter]

me too. that patch was attached at Comment No.10, but that comment was already 
deleted.

Original comment by Supe...@gmail.com on 27 Mar 2010 at 7:49

[GoogleCodeExporter]

The patch was wrong, so we had to take it down.
Necreia says he's working on it still.

Original comment by ramapcsx2 on 27 Mar 2010 at 2:34

[GoogleCodeExporter]

Yep.  Fix caused regressions, so working on it again.

Original comment by necr...@gmail.com on 28 Mar 2010 at 10:07

[GoogleCodeExporter]

Fixed, please test.  I haven't been able to find regressions.

It'll also speed up games with 16-bit graphics, though they are typically not 
slow to
begin  with.

Original comment by necr...@gmail.com on 30 May 2010 at 4:56

[GoogleCodeExporter]

Tested successfully in Mana Khemia 1 (Alchemists of Al-Revis)

No longer have random black lines overlaying the surface of characters in 
battle.

Tested With GSdx Settings' Renderer:
Direct3D9 (Hardware)
Direct3D10/11 (Hardware)

Waiting for moderators whether the patch can be merged into trunk or will be 
always
treated as a user contributed patch?

Original comment by Supe...@gmail.com on 30 May 2010 at 11:45

[GoogleCodeExporter]

Reviewing, and wb, necreia :)

Original comment by ramapcsx2 on 30 May 2010 at 1:30

[GoogleCodeExporter]

Hmm, not seeing any visual regressions, but a few games crash with this.
(Digital Devil Saga for example) :(

Original comment by ramapcsx2 on 30 May 2010 at 1:56

[GoogleCodeExporter]

Will the patch probably become a gamefix when it cannot be merged into trunk?

Original comment by Supe...@gmail.com on 30 May 2010 at 2:34

[GoogleCodeExporter]

@rama - Thanks.  I don't see how it could crash a game, hmm.  Any others?  I 
don't
have DDS to test with

@super - Ideally part of trunk since it seems like the logical thing to do for 
all
games, but if it really does crash other stuff it'll need to be a gamefix or
game-based 'hack' in the plugin.

Original comment by necr...@gmail.com on 30 May 2010 at 4:00

[GoogleCodeExporter]

Found the issue -- thanks for the testin' Rama

Additionally, deleted the first to avoid confusion.

Original comment by necr...@gmail.com on 30 May 2010 at 4:11

[GoogleCodeExporter]

After some more fixes, here's an updated version.

Original comment by necr...@gmail.com on 30 May 2010 at 9:40

Attachments:

• Fix-16bit-ATST.patch

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Glad to see this is getting fixed, though i assume the .patch file requires me 
to 
compile my own build with it(i know it can't be loaded into "patches" in pcsx2 
since 
that takes .pnatch), and that I do not have the skill/know how to do. Any 
clues as to when this will be included in future revisions?(i know a website 
that 
compiles the latest revisions)

Original comment by infelp...@hotmail.com on 7 Jun 2010 at 9:12

[GoogleCodeExporter]

Uh, yeah, this isn't getting fixed just yet.  We talked over the patch on IRC 
and
it's still wrong.

Original comment by sudonim1 on 7 Jun 2010 at 6:55

[GoogleCodeExporter]

I wonder I wonder... if this is supposedly 1 bit alpha, why is everything 
FMT_32 then? O.o

Original comment by KrossX3 on 13 Feb 2012 at 10:43

[GoogleCodeExporter]

If I force the texture format to TEX0.CPSM, it actually works quite well. Well 
meaning that sprites are actually FMT_16 while the rest is zero (32). Then just 
messing with that alpha in that section of the shaders works nicely without 
messing the HUD.

So something's wrong with that format selection me thinks.

Original comment by KrossX3 on 13 Feb 2012 at 11:24

[GoogleCodeExporter]

I think this is about it. A nice optional hack.

Original comment by KrossX3 on 19 Feb 2012 at 8:46

Attachments:

• Sprite_Alpha_Hack.patch

[GoogleCodeExporter]

Committed in r5101. Keeping the issue as the bug still should be fixed :p

Original comment by ramapcsx2 on 22 Feb 2012 at 10:45

[GoogleCodeExporter]

Fair enough. Although I dunno if it's a bug, as opposed to just a side effect 
of using different hardware. =S

Original comment by KrossX3 on 22 Feb 2012 at 11:11

[GoogleCodeExporter]

Updated hack, with 3 stages checkbox. 

Full would not check for the CPSM format (for Tales of Destiny) so it's the 
more aggresive option, and the middle option being the current hack.

Original comment by KrossX3 on 25 Feb 2012 at 11:22

Attachments:

• Update_Spritehack.patch

[GoogleCodeExporter]

Hmm, why the change in GS.h?

Original comment by ramapcsx2 on 25 Feb 2012 at 7:01

[GoogleCodeExporter]

Oh is just the same but in hex, and the correspoding comment to see which are 
also CPSM formats. That was just because I log this with hex, and since 
hovering would give me the decimal anyway I just changed to have a hex 
reference handy.

No functionality change at all, can just be ignored.

Original comment by KrossX3 on 25 Feb 2012 at 7:06

[GoogleCodeExporter]

This one seems to be better: http://pastebin.com/WAyYM0JZ

Sadly, it makes the "nice blending" thing useless. And it only works ok for the 
cases triggered by the hack (which is good in this case I suppose).

I checked this, because when I PIX'd some of the gsdumps there's some black 
thing drawn on top of the sprite right after the sprite itself has been drawn 
(to make their borders look soft perhaps) and it's the actual source of the 
evil black lines. Doing nothing in that case avoids the inner lines while 
keeping the border blackness. A precision thing perhaps... ?

Original comment by KrossX3 on 1 Mar 2012 at 2:40

[GoogleCodeExporter]

Progress on this.  Or maybe I've just rediscovered what I used to know.  In any 
case, I should document it here so I don't have to find it again.

The engine draws feathered sprites with a greater than or equal to depth test 
and an alpha test which passes pixels more than x% opaque and sets AFAIL to 
FB_ONLY.  The purpose of this is to drawing over transparent borders (they 
don't update the depth buffer) so that only the outer borders will be left.

This doesn't currently happen in gsdx because there is no way to perform a 
depth test but not update the depth buffer on a pixel basis.  So we perform two 
passes, first sucess pass, which updates both the framebuffer and the depth 
buffer, then we do the failure pass, which updates only the framebuffer.  
Because the failure pass consists of only edges, all the inner edges will be 
drawn on top of the sprite.

Now for fixing this, in the FB_ONLY AFAIL case it's probably safer to do the 
failure pass before the success pass, though neither is perfect.  Because we're 
seeing inner edges, this means that the sprite components are on the same depth 
plane and drawing order must be important.  But drawing one sprite at a time is 
extremely costly, and I can't think of a reasonable technique using 
AFAIL=FB_ONLY where it would be desirable for the failures to be drawn over the 
successes rather than the other way round.

Changing this in the function as it is now would make a convoluted function 
more convoluted unfortunately.  Then again more thought might be required, 
failure pass first for AFAIL=FB_ONLY might be undesirable.

Original comment by sudonim1 on 21 Jul 2012 at 3:56

[GoogleCodeExporter]

Some of what I said in the second paragraph is wrong (and missing a word), 
since I later realised that the drawing order must be important.

Original comment by sudonim1 on 21 Jul 2012 at 4:36