handling of recently suspended cameras

AlicVB commented 2 years ago

In mid-december, lot of cameras without samples in raw.pixl.us have been "suspended" by rawspeed.

Although I perfectly understand the need of those samples, the way it has been done is quite difficult to understand for the average users, especially because they suddenly can't open and edit or export their raw files... even those that have been edited before... This is a great breach in the the confidence pact between users and the application (from an external POV we just "trash" their work). How can we explain then, that we take care of their data, that we will keep them safe, that they can upgrade safely to new versions, that they don't need to mass convert to dng, ...

Of course we can argue that the support is just "suspended" until they will provide sample to raw.pixl.us, and they will do it... but the "until" part is not true : they will have to wait for spare time of devs, maintainers, publication of a new version... This is just annoying for hobbyist, but that's a total no-go for professionals!

What I would propose to solve this very serious issue is the following:

instead of disabling completely the support, rawspeed return a special flag to indicate to the main program that the file is problematic.
then it's up to the main program to handle that correctly, for example :
if the raw file has already been edited => show a "no-too-intrusive" and very explicit alert but still decode the image as before
if the raw file is "new" => show a more intrusive alert but still decode the image
In all cases we do add a reference to the Web site for the RAW contribution

That way : => we inform end users very clearly of the problem (not in some obscure release note) => we don't "trash" their work

How does that sound ?

LebedevRI commented 2 years ago

Hello.

Let me note that for the last five years, every single darktable release notes had this disclaimer:

Important note: to make sure that darktable can keep on supporting the raw file format for your camera, please read this post on how/what raw samples you can contribute to ensure that we have the full raw sample set for your camera under CC0 license!

The users were warned, for the last five years, that this would happen, and it finally happened.

professionals

Presumably they don't rely on a bleeding version of whatever software, and they pay the developers for support, so time won't be a problem for them.

TurboGit commented 2 years ago

I also vote for a warning and be able to keep the old edits. We cannot break user's work like that. Note that most users don't know what the relation between all those projects (darktable, rawspeed, exiv2, libraw, opencl...) so from their point of view we have broken their work and invested time into their pictures.

To me, @AlicVB proposal is really a sound one.

AlicVB commented 2 years ago

@LebedevRI : the problem in real life is that most users don't read release notes. I deplore that, but it's a fact. On the other hand, I don't read either most of release notes of the different apps I use on a daily basis, except if I'm waiting for something specific...

We may say," too bad for them. Next time they will..." OK, why not. But NOT for the old edits... Imagine what would happen if we simply drop the old equalizer module... after all it has been deprecated for years now, users have had enough time to upgrade their edits...

Now let's face our responsibilities : we (at dt) have failed to properly inform and warn users. We should have implemented more obvious "in-app" warnings. The question is : what can we do now ? imo we can't let the situation like this !

Hence my proposal...

AlicVB commented 2 years ago

@LebedevRI : another recurrent issue we face in the different messages we have, is that lot of the "suspend" cameras are old ones that user don't own any more. They can (and they have done for at least 2 cases) upload samples of their old raw files, but only for the aspect ratio / compression / ... they have used. (most users only use 1 sort of raw and don't change them) So we won't have the full set of raw files for this cameras. It would be good to allow the possibility to unlock the support in this cases !

parafin commented 2 years ago

I would say removing support for cameras in this circumstances was fine, users can continue to use older darktable release until support is brought back. It definitely shouldn’t affect “professional” work, since one should test any updates if he relies on the software for work, and automatic backups allow going back.

The only issue is IMHO the mentioned fact, that few people read release notes, so warning should have been more prominent in darktable UI itself. Which, I might note, is not in rawspeed’s area of reach. But since this already happened, I don’t really see the point of discussing it now, it’s not like we expect any such downgrade to happen again. This issue should have been raised years ago, now it’s of no consequence.

AlicVB commented 2 years ago

This issue should have been raised years ago, now it’s of no consequence.

I sort of disagree (sorry) If we were sure to be able to put back the support quite quickly, at least for people of goodwill, that would be ok, but will this happen ? As said before, this people, once the problem raised, will (probably) upload some raw (I know at least 2 case, where it happen) but for the 2 mentioned cases, only 1 raw format/compression/... has been uploaded, just because the users only has this sort of files in its archive and don't own the camera anymore... And currently, without the full set of file, the support will not be restored (iiuc) That mean that we let those users with an old unmaintained (even for critical/security issues) version, without any mid-term hope of a solution...

That said, you raise a good point in the fact that its not completely rawspeed fault here. It would have been up to dt to warn users 5 years ago, directly in the UI when opening each of this raw files. Main problem is the breakage of old edits. we simply can't afford that. and we can't put that back without some rawspeed changes...

LebedevRI commented 2 years ago

I have three questions:

Will you be implementing said noisy pop-up warning in darktable the moment this functionality is provided by rawspeed?
Will you immediately backport said warning into the very next darktable point release, which i guess is going to happen soon?
I'm not comfortable with providing such functionality, especially without expiration date. I'd be okay with keeping it until the next darktable major release, or at most the one after that. Does that sound like an acceptable timeframe?

AlicVB commented 2 years ago

@LebedevRI : Thanks a lot for your answer.

I'm not alone here, but for me the answer for 1 and 2 is YES (That's the main idea) For point 3 (the main one) the answer is more difficult. Ideally I would say that we should kept it forever for "already edited" raws. On the other hand, rawspeed has no way to know that point... and I understand how that can be uncomfortable for rawspeed ...

Correct me if I'm wrong, but I suppose that the main issue with that is that you'll not be able to change/add some functionalities in rawspeed and tests them on this files, right ?

Anyway, keeping it until next december would allow us to have some time to decide how we can handle that backward compatibility in dt. And hopefully until then the list of problematic camera will decrease a lot. That's our main common goal here !

TurboGit commented 2 years ago

@LebedevRI : As for @AlicVB this is yes and yes for point 1 and 2 above. So if you can give us a flag for those cases we can work on this and have this ready for 3.8.1. Thanks.

LebedevRI commented 2 years ago

I'm afraid point 3 is the main blocker here :) There needs to be a clear timeline, "forever" simply doesn't work. I guess i'd be okay with keeping this for at most one more darktable major version (3.10? that is, revert before 3.12)

TurboGit commented 2 years ago

@LebedevRI : Let's try one full year as it is done when we have deprecated messages in dt for a module. That means 2 releases, as next release is 4.0 in mid-2022, we can say that it will be fully deprecated in 4.2 (Dec 2022). Is that ok with you?

AlicVB commented 2 years ago

@LebedevRI : I've seen your patches/PR (even if I'm not sure follow all them) and I'd like to thank you for that. Now we have to work on the dt side to handle that new flag and warn very explicitly the users ! That should be easy and done quickly, certainly before 3.8.1

for my other question. iiuc, you currently need the full sample set of the raw files to re-enable the support, right ? Do you think it would feasible to accept (for those cameras specifically) only some samples and only allow those formats to be re-enabled. For example we have a french user who has face the problem for its old DMC-G2 raw files. Once he has understood the origin of the problem, he has immediately uploaded the only format he has in its library : 4:3. But as he doesn't own the camera anymore, he has no way to upload the other formats...

LebedevRI commented 2 years ago

Ok, your turn.

Do you think it would feasible to accept (for those cameras specifically) only some samples and only allow those formats to be re-enabled.

That may be fine i guess.

TurboGit commented 2 years ago

@LebedevRI : Done on dt side, thanks you for your support. We have a log message plus a persistent message with a red background, not beautiful but no one will be able to say I didn't know :)

LebedevRI commented 2 years ago

Note that it will only catch the cases where people actually load such images during the time that message will be there. I guess that message should have been added 5 years ago, shouldn't it have...

TurboGit commented 2 years ago

I guess that message should have been added 5 years ago, shouldn't it have...

Probably, but it has not been done and here we are now. Better now than never, no?

LebedevRI commented 2 years ago

Sure.

darktable-org / rawspeed

handling of recently suspended cameras #335