Canon CR3 support. - Githubissues

LebedevRI commented 6 years ago

Though, until the production camera is actually released, and the full set of freely-licensed samples is contributed to the https://raw.pixls.us/, it won't be merged.

lclevy commented 5 years ago

You'll have also information here: https://github.com/lclevy/canon_cr3/blob/master/readme.md#samples

⁣Envoyé par Blue

Le 3 janv. 2019 à 21:53, à 21:53, Chema notifications@github.com a écrit:

@LebedevRI tomorrow you have it ;)

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/darktable-org/rawspeed/issues/121#issuecomment-451273680

LebedevRI commented 5 years ago

@LebedevRI tomorrow you have it ;)

the upload form will say that there are samples already, do upload everything anyway.

LebedevRI commented 5 years ago

You'll have also information here: https://github.com/lclevy/canon_cr3/blob/master/readme.md#samples

Yes, i'm aware of those samples.

they are not on RPU, therefore i don't/can't/won't use them in the integration testing of the library.
They are not under CC0, therefore their usage is somewhat questionable
We don't have CC0 sample set anyway, so we need one anyway

neo22s commented 5 years ago

Hello,

All RAW files here properly named: https://mega.nz/#!NsQRgCxT!vCh4EaU8FfPsXCl2ge4Ar4XGMhbgtjPMu8qBi6Gp2q0

I am uploading 1 by 1 but It gets stuck and I need to leave in 10 minutes.

LebedevRI commented 5 years ago

No worries, please do finish uploading them when/if you have time. Sadly, this isn't really time-critical issue here..

neo22s commented 5 years ago

Done! so what's next? :D

I really need this.. I am trying to make a living from photography and doing it only using Linux gets hard... ;)

LebedevRI commented 5 years ago

Sadly, the best current advice i can get is to build a chroot (https://wiki.debian.org/Debootstrap e.g.), install wine there, and temporarily use adobe dng converter to temporarily convert the raws to DNG, while keeping the originals.

neo22s commented 5 years ago

Ouch. Terrible workflow.

When do you think we can have this done?

On Sat, Jan 5, 2019, 14:25 Roman Lebedev <notifications@github.com wrote:

Sadly, the best current advice i can get is to build a chroot ( https://wiki.debian.org/Debootstrap e.g.), install wine there, and temporarily use adobe dng converter to temporarily convert the raws to DNG, while keeping the originals.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darktable-org/rawspeed/issues/121#issuecomment-451655119, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOwPRn8FY0m82ps5LdcsueWXFavBW9Yks5vAKe3gaJpZM4SaQUi .

seanmavley commented 5 years ago

@neo22s I have the M50, and I must say working with it has been a challenge. Only adobe and canon's dpp support it. And both apps work on Windows only. No linux support.

Currently, my workflow involves running canon dpp in Windows, export all the files, then continue with darktable or gimp in Linux.

@LebedevRI I have the m50, in case samples would be needed from it. Over 5k of CRAW files to pick from in almost all forms of lighting conditions.

neo22s commented 5 years ago

@seanmavley

Luckily for me I was barely using the RAW files, I do fashion photography and only if I messed up badly I needed to use the RAW files (wrong white balance or really bad exposed).

If I was a wedding photographer (which I do sometimes) will be a nightmare. I am ready to use windows those days if this does not go faster.

Such a pity :(

jlbee commented 5 years ago

Any news on this? I use the EOS R, and can't open any of my RAW files. It kind of makes me regret trading in my previous camera since it had really good support. I submitted my pics to a repository I was linked to on Mastodon in December. I can't get Adobe or Canon's software to work on anything, and my desktop is too old to download anything to. I can do basic stuff in GIMP, but it just isn't the same.

LebedevRI commented 5 years ago

I have not looked further into that yet. If anyone wants to, feel free to work on it.

lclevy commented 5 years ago

it seems Alexey is working on it https://github.com/lclevy/canon_cr3/issues/7

LebedevRI commented 5 years ago

Nice! Good spec will be better than working code (looking at you, fuji decompressor).

neo22s commented 5 years ago

Thanks guys 👌

On Tue, Feb 5, 2019, 22:37 Roman Lebedev <notifications@github.com wrote:

Nice! Good spec will be better than working code (looking at you, fuji decompressor).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/darktable-org/rawspeed/issues/121#issuecomment-460813761, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOwPXsVkEafeNjogMPsRCZtfV2a9AVSks5vKfmygaJpZM4SaQUi .

Alexey-Danilchenko commented 5 years ago

@LebedevRI - yet working code just works and is miles better than empty statements. And if you really expect to have good specs in a world of proprietary raw formats, I think it is time to wake up and have a reality check

LebedevRI commented 5 years ago

A working code (source code under open license, not something else) is sure better than nothing.

That fuji remark was mostly in context of

lclevy/canon_cr3#7

which seemed like an improvement/contribution to the attempt to spec-ify the codec. I omitted word "even" in "Good spec will be better", thus, having an idea of the state of fuji decompressor, it does read as some variation on "better no code than that again".

I admit, i'm fond of the effort fuji decompressor took, and i will freely admit that i'm not fully fond of the structure of that code.

In particular, i'm observing that the threading is per-input-tile, and there are 4..12 of them, which results in load imbalance. so, i'we been wondering whether the algo can be separated into stages, so some of decoding can be more fine-grainedly parallelized (per-row), but so far i was unable to fully comb through it. Although i haven't really looked into that too deep.

Oh well, hopefully this didn't degrade things even further :)

Thank you for looking into it!

Alexey-Danilchenko commented 5 years ago

That format came from reversing just as in Fuji case header format for their data. Neither will give you the specifications of original format or decoding algorithms. So whatever that is - it is definitely not and attempt to specify the codec, only the file format.

As with anything in reversing, unless you will want to put an effort in understanding the algorithms and rewrite the code, it will not improve from how it was coded by original developers (Fuji, Canon etc.) and from what I seen their coding is left to be desired. I certainly won't do it for you (I have enough spare time spent on it already) so perhaps it's task you will take upon yourself (it will certainly come out better than speeches). In the meantime, I will release and license CR3 (when ready) to libraw only.

LebedevRI commented 5 years ago

To be honest i'm not quite sure what prompted such harsh responses. It was logical that fuji decompressor code was not produced from a good human-readable spec.

That format came from reversing just as in Fuji case header format for their data.

Confirms that.

I have not (nor have i wanted to) insulted you, your coding/developing abilities, reversing abilities. I have not demanded/requested/etc that you improve that reversed code. I have only voiced my frustration with that current implementation, with how inefficient it is, and voiced (false, apparently) hopes that in CR3 case, the approach would be different.

neo22s commented 5 years ago

Hello!!

Any news about this? thanks :D

kznsq commented 5 years ago

Hello! The patent is describing the prediction coding with K parameter (Golomb-Rice coding base value), which is actually a power of 2. Each step this K parameter is updating. But where is a initial value? Maybe here? "19 bits value (3 right most bits of byte#9 + 16 bits at offset 10). Substracted to size at offset 4. Likely to compute exact meanful bits at end of the encoded stream, as size is rounded to 8 bits (observed values are 0 to 7). y" https://github.com/lclevy/canon_cr3

lclevy commented 5 years ago

if you use parse_cr3.py -v 3, the value is displayed. I see values 0, 1, 2, 3, 4, 5, 6, or 7.

a discussion with a wavelet expert told me each level add 2 bits. "With this logic each stage adds two bits, so level 1: 14-bit input -> 14+2 bit output, which is used for level 2 level 2: 16-bit input -> 16+2 bit output, which is used for level 3 level 3: 18-bit input -> 18+2 bit output". so yes, these 19 bits might be related to 20 bits output of subband level 3

kznsq commented 5 years ago

As I can see, the encoder uses simple formula: (unaryCodedValue<<K)+KBits "0063 Step 1: 0s equal to a number indicated by a value obtained by shifting the symbol S (binary expression) to the right by Kbits are arranged, and binary data is generated by adding 1 after these 0s. 0064 Step 2: Lower K bits of the symbol S are extracted and added after the binary data generated in step 1. "

Why there are 21 bits? "00000000 00000000 00000000 00000000 00000000 001 = 42

00000 00100111 10100101 = 10149 (next 21th bits)"

lclevy commented 5 years ago

please contact me lclevy at free dot fr, it will be easier

neo22s commented 5 years ago

Hello,

Sorry been so annoying, but my current workflow sucks. Although I am getting used to not use RAW files unless necessary.

There's any update on this? thanks a lot.

kenthinson commented 5 years ago

@neo22s have you tried wine + Adobe dng converter? Working well for me.

I am curious about this projects progress too though :)

seanmavley commented 5 years ago

It's been a year already since cr3 came in. I moved on for now. I just run Windows 10 in Virtualbox these days, with Canon DPP, and Lightroom. cc @neo22s

neo22s commented 5 years ago

Hello

@kenthinson could you share with us the workflow you are using and how does it work?

@seanmavley I am using an old laptop I have with windows.... but I hate the workflow soooo much damn.

I am an advocate for FOSS and this is so frustrating since I am trying to convince other pro photographers to move to linux but if they find this kind of barriers.... such a pity

What can we do? maybe contact canon with many signatures to release the specifications?

neo22s commented 5 years ago

@kenthinson got it and working pretty well! but I will not convert all RAW only of those that I need to work with.

Steps: 1- Install wine 2- Download https://helpx.adobe.com/photoshop/using/adobe-dng-converter.html for windows 3- Open adobe DNG 4- Select folder, click convert 5- Open digikam, or darktable. All working!

works quite fast and seems stable, thanks!

clanmills commented 5 years ago

The idea of contacting Canon with many signatures (a petition) may be effective. The case for Canon to participate in OSS is a subject for marketing and management.

My thoughts are: 1) We need Canon to believe that putting the Camera SDK into the public domain gives them business "advantage" over their rivals.

2) If a major "player" such as Adobe, Microsoft or Google requested this as a way of reducing development effort for both the "player" and Canon.

3) The reason they don't participate in OSS is historical. In the 1990s, they developed relationships with "players". OSS wasn't significant to their business and they (correctly) perceived OSS as "lots of pain for little gain".

4) Since the 1990s, the world has changed. Think about Microsoft. They now participate and cooperate with OSS. We'd like Canon to rethink their position as we move into the 2020s.

Perhaps this matter should be discussed at LGM next month. We probably need a "Champion" to lead the effort to set up a meeting with Canon to discuss this. I'll be be happy to assist the "Champion".

https://libregraphicsmeeting.org/2019/

neo22s commented 5 years ago

@clanmills that sounds like a great idea!

kznsq commented 5 years ago

I hope the CR3 format will supported by LibRaw soon: https://www.fastrawviewer.com/blog/FastRawViewer-1-5-1-Beta-CR3-Support?fbclid=IwAR21w54ZtZid9bHO_M983PFUbBXjJroPI5vcmk-AZwzjgNG0Kqb98VRcQfc

neo22s commented 5 years ago

But does not work on Linux....

On Mon, Apr 15, 2019, 10:09 kznsq notifications@github.com wrote:

I hope the CR3 format will supported by LibRaw soon:

https://www.fastrawviewer.com/blog/FastRawViewer-1-5-1-Beta-CR3-Support?fbclid=IwAR21w54ZtZid9bHO_M983PFUbBXjJroPI5vcmk-AZwzjgNG0Kqb98VRcQfc

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/darktable-org/rawspeed/issues/121#issuecomment-483151155, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOwPfe2t3Aj5U0XamH5Fy3xbtrsJBnmks5vhDOtgaJpZM4SaQUi .

parafin commented 5 years ago

There might be other reasons for not open-sourcing their code, for example Canon might be using 3rd-party code for which they don't have the right to release the source code. Or there's some know-how they don't want public yet. So you should keep in mind, that this decision might not be as trivial to make for Canon as you picture it.

clanmills commented 5 years ago

@parafin You are quite right that there can be very good reasons what we do not know. However it makes sense to "make the case" for support for OSS. If Canon believe it's in their best interest to support OSS, they will investigate and solve consequential puzzles.

However, to persuade Canon (or any manufacturer or "player") to support OSS is a huge challenge.

LebedevRI commented 5 years ago

I hope the CR3 format will supported by LibRaw soon:

I think the most likely path will be:

libraw implements it
- they will either go hostile and release it not under free license, then i don't know
- or it will be under free license
I will either integrate it here as is (best to try to have some baseline for correctness/performance), and re-rewrite it from there;
or somehow grok whatever code they end up producing, condensing it into some written form, and writing all the code from scratch. could work, may avoid license issues should they not release under free license; but this certainly won't help with baseline.

That being said, please stop adding comments "what's the status here". I'm going to hide some comments here..

felixkkkk commented 5 years ago

Hi, I hope this isn't considered "what's the status here", but I have a question that touches this aspect ;-)

I stepped up from 5D mk ii to EOS R and used to shoot RAW only, as darktables interpretation of the RAWs was close enough to the JPGs and Canon colors, and I really liked the way they came out of darktable.

When I got my EOS R, I first wanted to return it immedeatly, as darktable dind't open the CR3s. After a bit of research I dug up Adobes DNG-converter, which runs under wine. Now darktable can handle the files, but colors, contrast, everything is way of and of course I can tweak single pictures to look kind of ok, but not a batch with copy and paste. No "default preset", thats okish...

Now the question: Is this going to change for good again, when darktable supports the EOS R ?

This is really important to me, as I'm really unhappy with the camera at the moment, which is a pity.

Hope this question is ok... best regards and thanks!

seanmavley commented 5 years ago

@felixkick

This is really important to me, as I'm really unhappy with the camera at the moment, which is a pity.

Your not being able to open the CR3 format in Darktable is no fault of Canon or the camera so by saying you're "unhappy" with the camera is like saying you bought an electric car and your nearby fuel station don't have electric charging so you're unhappy with the car.

~Anyway, Libraw appears to be what Darktable and the other open source programs use for Canon raws.~ As at the writing of this message, Libraw doesn't have CR3 support.

https://discuss.pixls.us/t/my-take-on-rawtherapee/13885/60?u=seanmavley

So, let's hope for the support to be included soon.

The EOS R is a great camera. I have similar problem as user of Canon M50. I use Lightroom or DPP.

LibRaw commented 5 years ago

AFAIK, Darktable does not use LibRaw

neo22s commented 5 years ago

Hi, I hope this isn't considered "what's the status here", but I have a question that touches this aspect ;-)

I stepped up from 5D mk ii to EOS R and used to shoot RAW only, as darktables interpretation of the RAWs was close enough to the JPGs and Canon colors, and I really liked the way they came out of darktable.

When I got my EOS R, I first wanted to return it immedeatly, as darktable dind't open the CR3s. After a bit of research I dug up Adobes DNG-converter, which runs under wine. Now darktable can handle the files, but colors, contrast, everything is way of and of course I can tweak single pictures to look kind of ok, but not a batch with copy and paste. No "default preset", thats okish...

Now the question: Is this going to change for good again, when darktable supports the EOS R ?

This is really important to me, as I'm really unhappy with the camera at the moment, which is a pity.

Hope this question is ok... best regards and thanks!

Got same issue... many times unless I need to recover a lot highlights or shadows or change the WB I use the JPEG instead of the RAW file. I shoot always RAW + JPEG

Comparison JPEG straight out of camera IMG_5345-from-camera

And the RAW exported in Darktable with defaults (quality to 90% if not wast 28MB) IMG_5345_01

clanmills commented 5 years ago

I'm the maintainer of Exiv2 - the C++ Metadata Library. We don't support Canon CR3 and we don't have a schedule to implement this.

I'm much more empathetic to your unhappiness with Canon. Several folks have written to Canon to ask "will you make your CR3 SDK available to the open source community". And while Canon have replied promptly and politely, I don't believe they have. For sure, they've never contact Team Exiv2.

I believe Canon have an SDK for the CR3 and make it available to Adobe, Microsoft and other commercial software vendors. For CR3 to be a commercial success, it's essential that PhotoShop can operate on the files. Regrettably, while Canon recognise the important of software support for the format, they don't consider open-source support to be necessary.

This status is regrettable. It leaves Exiv2 (and other libraries) with the obligation to reverse engineer the format and write 100% of the code. In contrast to Adobe with 11,000+ people, Team Exiv2 has 3 part-time volunteers and a few friends who contribute quite frequently.

Now, I know you can say "well give it higher priority" and here's my response. We have 3 engineers. I handle almost all the user correspondence, documentation and releases. We currently have a program of quarterly "dot releases". Every three months, I make a new release with security and bug fixes. Exiv2 v0.27.3 is scheduled for 2019-09-30. My colleagues @D4N and @piponazo are undertaking refactoring the code into C++11. Exiv2 v0.28 will be the largest overhaul of the Exiv2 in its 16 year life.

CR3 is something I'd like to see added to Exiv2, however we don't have the resources to undertake this for release in a future "dot" and/or in Exiv2 v0.28 which will ship in 2020.

I'm a retired Adobe Computer Scientist and would like to retire from Exiv2 by the time of my 70th birthday on 2021-01-18. I'm currently writing a book "Image Metadata and Exiv2 Architecture" and will present this at a work-shop at the LGM conference in Rennes, France in May 2020. The book is my legacy in the hope that Exiv2 endures and is supported into the future. https://github.com/Exiv2/exiv2/issues/911

Being retired, I have time to spend on Exiv2. My colleagues have full-time jobs which must be given higher priority than their open-source activity. In fact, I've undertaken the release and user support roles to help them to focus on the Exiv2 v0.28 project. I haven't asked them to work on CR3.

If you know anybody willing implement CR3 support in Exiv2, I am happy to mentor and encourage somebody to undertake the challenge.

I also encourage you to write to Canon and ask them to make their CR3 SDK available to the open source community.

felixkkkk commented 5 years ago

Hi,

Your not being able to open the CR3 format in Darktable is no fault of Canon or the camera so by saying you're "unhappy" with the camera is like saying you bought an electric car and your nearby fuel station don't have electric charging so you're unhappy with the car.

fair enough - should have checked. Would have bought different camera or waited longer. I love darktable and have all my images in the database...

AFAIK, Darktable does not use LibRaw

That's what I think too, though libraw seems to get CR3 going soon: https://www.libraw.org/comment/5387#comment-5387 Hope they release the code, so darktable can follow!

Got same issue... many times unless I need to recover a lot highlights or shadows or change the WB I use the JPEG instead of the RAW file. I shoot always RAW + JPEG

But how do you get those pictures not to "drop out" of a set. They look different and distinguishable (at least, when I try...)

That's the point of my question, whether darktable will get those EOS R CR3s working nicely (as it did and does with my 5d mk ii) or maybe not. (And as a side question, what will happen to my DNG-edits, will they suddenly change, when the camera is supported?)

felixkkkk commented 5 years ago

I'm much more empathetic to your unhappiness with Canon.

Thanks...

Now, I know you can say "well give it higher priority" and here's my response. We have 3 engineers.

Wouldn't do that :-) I admire your and the whole open source communitys passion and efforts! Thanks!

I also encourage you to write to Canon and ask them to make their CR3 SDK available to the open source community.

Will do that! :-)

neo22s commented 5 years ago

But how do you get those pictures not to "drop out" of a set. They look different and distinguishable (at least, when I try...)

I am carefully setting my WB manually in each scene ;) I've done it always since I love to see the pictures directly from camera and send them directly to customers. Saves me a lot of time. If I mess it up then I need to process the RAW's which some times happens :/

I also encourage you to write to Canon and ask them to make their CR3 SDK available to the open source community.

I have done it. I have the same feeling working developing Open Source Software since more than 13 years now... I made it my full time job and part time professional photographer. Maybe we can make a money collect to pay someone to do it? I will not mind paying a bit from my own money and to see this released.

Anyone else will donate to make this happen?

clanmills commented 5 years ago

Google might pay a student to do this. There's a program "Google Summer of Code". You can read about it here: https://github.com/Exiv2/exiv2/issues/659

I would like to imagine that "crowd funding" is possible. I suspect the cost to hire a Silicon Valley Software Contractor for a few months is more than we could raise.

Maybe when I've finished the book and the workshop in France, I'll implement CR3 as my final coding project. If Canon provide the CR3 SDK, the job might be quite easy as they will provide the code to parse and write the format. However, they might impose NDA conditions that we cannot fulfil.

D4N commented 5 years ago

I have done it. I have the same feeling working developing Open Source Software since more than 13 years now... I made it my full time job and part time professional photographer. Maybe we can make a money collect to pay someone to do it? I will not mind paying a bit from my own money and to see this released.

Anyone else will donate to make this happen?

There is https://www.bountysource.com/, but beside some prominent and monstrous issues, most of the bounties raised are not enough to make it worth working on those issues from a financial perspective.

I would like to imagine that "crowd funding" is possible. I suspect the cost to hire a Silicon Valley Software Contractor for a few months is more than we could raise.

Yeah, hiring a contractor is going to cost absurd amounts of money, especially if they are from the Silicon Valley.

lclevy commented 5 years ago

Hi,

Alexey Danilchenko contributed code to libraw for CR3, so there is already "open source" support for it.

Kind regards,

Laurent

clanmills commented 5 years ago

Thanks @lclevy It's very helpful to hear that about this.

seanmavley commented 5 years ago

In that case, @lclevy is there anyhow that can be used in any software on a Linux?

drazde commented 5 years ago

@lclevy Is there a possibility to try and test?

darktable-org / rawspeed

Canon CR3 support. #121