Use a tag-based model for configuration files

[GoogleCodeExporter]

What steps will reproduce the problem?
1. User installs version x of 400plus.
2. User saves some settings and / or presets to the card.
3. User installs version y > x of 400plus.
4. User cannot load settings or presets from the card.

What is the expected output?
User should be able to load configuration files from previous versions, even if 
new parameters have been added to the settings.

What do you see instead?
Camera acts as if no configuration files exist at all.

Please use labels and text to provide additional information.
We should change to a tag-based model for the configuration files (see TIFF 
specification for an example of what I mean): instead of dumping memory areas, 
we should write pairs of {PARAMETER, VALUE} to the files; this way, we can 
later load that file using more recent version, even if new parameters have 
been added.

Original issue reported on code.google.com by eduardo....@gmail.com on 28 Apr 2011 at 6:27

[GoogleCodeExporter]

If you were to choose an .XML or standard .INI type key-value-pair type file - 
or even something else with a known published interface - I might be interested 
in developing a 400plus GUI executable to run from a PC to allow to create, 
edit, reset, remove configuration and preset file data on the CF card.  Now the 
user would have the choice of using the camera, or the PC app to change 
configuration and/or preset settings.  This could have several benefits 
including simplifying the request to have "named" Presets other than Preset 1, 
Preset 2, etc.

I would write this in C# and take a shot at making it cross platform with MONO.

Original comment by mike.guf...@gmail.com on 28 Apr 2011 at 4:16

[GoogleCodeExporter]

I have been taking about this with 0xAF: he also proposed to use a text-based 
approach. I still have my reservation about letting users tinker with the 
files, but I do not want to become a roadblock. So... text files welcome.

Original comment by eduardo....@gmail.com on 28 Apr 2011 at 8:29

[GoogleCodeExporter]

I actually tend to agree with you about using free and clear text which is very 
easily breakable by suspicious or unknowing users (no telling what kind of 
issues this might cause with the camera).  

Another idea would be to keep the concept of a familiar key-value or tag based 
file type and use a very simple cipher algorithm on the file for encryption 
(Vigenère cipher, Caesar cipher, etc.).  This would provide a bit more 
protection without adding a lot more code...

Or we could just stick with binary as long as the format is documented well and 
contains versioning information, etc.

Original comment by mike.guf...@gmail.com on 28 Apr 2011 at 9:25

[GoogleCodeExporter]

if we need to obfuscate the file, XOR will be enough... no need of encryption.
after all it's an opensource... whatever encryption we put will be visible and 
reversed.
and there is no point to obfuscate the file.

text files are good... and we will have ranges for the inputs...
if its an int input, we can have min an max, so whatever the user puts for the 
value wont be a problem
for the string inputs we can have max length... this wont be a problem.

i believe most of the ppl will edit the text config file only for setting the 
preset names.

i was going to do a text parser, but then i saw the ML solution for the config 
files, i like it. 
https://bitbucket.org/hudson/magic-lantern/src/9f843c096fda/magiclantern.cfg

Original comment by fired...@gmail.com on 29 Apr 2011 at 6:53

[GoogleCodeExporter]

The only reason that I can think to use text files is precisely to let users 
edit them; I do not understand the intention of encryption / checksums to 
prevent users from editing the files, then. If we are not going to let users 
edit the files, then tags are much easier to code.

And if we are going to use text only to rename presets, then I would use tags 
for the rest, wait until we have the option to do the renaming in the camera 
(we are going to need it, anyway), and then decide whether it is worth the 
effort.

Original comment by eduardo....@gmail.com on 29 Apr 2011 at 10:57

[GoogleCodeExporter]

<i believe most of the ppl will edit the text config file only for setting the 
preset names.>

I agree totally - MOST (just about all) people will use the config / preset 
files in the manner you have suggested.  Not me however, I will have multiple 
sets of config and preset files across a range of CF cards.  And make most of 
my changes within the files themselves rather than making all the settings by 
using the camera each time - whether it be using a free editor, or a more 
controlled home-grown GUI app from a PC.

But I admit I am not in the norm - never have been.  I like either idea 
(.tiff-like tags, or .ini-like text), whichever makes the most sense to code 
and maintain.  I guess there is time for more talk of this later on as it gets 
nearer to implementation.

Original comment by mike.guf...@gmail.com on 29 Apr 2011 at 5:54

[GoogleCodeExporter]

So, after Edu made the string editor I guess we can go with the binary files 
for now ?
At least until we have a real need of the text files.
Maybe it would be good to let the user edit the parameters of the presets 
offline, but maybe here comes the PC program. This way the input can be checked 
before written to the file.
Mike keep in mind that if you go for such program, you will have to keep up 
with version increments (and structural changes) in the binary file, at least 
once in a while. (would be nice to be OS portable too)

Original comment by fired...@gmail.com on 30 Apr 2011 at 8:42

[GoogleCodeExporter]

Ok, so now that the presets' renaming feature is done, I will explain my 
position about the config files again:

* Use a tag-based binary format for the files that users are not supposed to 
edit themselves.

* Use a text-based (unsigned and unencrypted) format for the files that users 
are supposed to edit.

We have currently three files:
* The SETTINGS file (stores the same information that the settings menu).
* The PRESET_x file (each one stores one preset, and contains a SETTINGS file 
inside).
* The PRESETS file (stores the names and positions of the settings).

Now, my question is: which of these files should be user-editable? what 
information is useful to the users?

Original comment by eduardo....@gmail.com on 30 Apr 2011 at 8:47

[GoogleCodeExporter]

As for the PC program:

Yes, I understand - synchronization amongst new settings/features added to the 
config file(s) as well as structure changes, and the ability so support older 
as well as newer versions of the file(s) would be a challenge.  But one that I 
think I am up to and have tackled previously in such projects.

C# does not afford me the ability to directly import the 400plus header (.h) 
files, but I think we can come up with something share-able within the code 
repository.  And by using #Develop and MONO 
(http://www.icsharpcode.net/OpenSource/SD/Features.aspx) - GUI support is 
already available for Windows/Linux/Mac users which would give us a broad 
audience to use such an app.

I am still tossing around this whole idea, and trying to generate requirements 
in my head.  If do-able, I believe it would be a valuable tool - especially if 
it could somehow support a CF card either IN-CAMERA or via an external or 
internal card reader.  How great would that be!

I'm getting a little long-winded here sorry, my next comment on this issue will 
discuss my thoughts for the file formats.

Original comment by mike.guf...@gmail.com on 1 May 2011 at 5:55

[GoogleCodeExporter]

My idea is that all of the config files would be of the same file format.

Formats I would tend to think would be useful (not in any order) are:

1) Text: of proven structure (.ini, .xml, .csv, etc.)
2) Text: custom structured (400plus custom format)
3) Binary: of known or custom structure (direct binary format, not tag based)
4) Binary: of known or custom structure (tag based such as TIFF, NBT, etc.)
5) Embedded single file database (SQLite or other small embedded Database) 

5,4,1 seem to me to be the best options.

My favorite for not easily editable by user would be 5)SQLite, then 4)TIFF or 
NBT.
My favorite for easily editable by user would be 1).ini like Linux .config 
files.

I think all are do-able, and don't really yet know how to decide.  The three 
(3) separate files could continue to be used as they are now, or they could be 
combined into one (1) file.  My opinion would be to at least keep the 
"Settings" file, and the "Preset or Preset(s)" file(s) separate for easy 
archiving and copying to multiple CF cards.  Sure would be nice to figure out 
how to partition the CF card so that the file(s) are not seen in root of CF and 
survive a format, but that is an entirely new topic to discuss and not 
appropriate for now.

SQLite info:
http://www.sqlite.org/whentouse.html
http://en.wikipedia.org/wiki/Embedded_database  (scroll to SQLite section)

Original comment by mike.guf...@gmail.com on 1 May 2011 at 6:41

[GoogleCodeExporter]

about the PC program, let's discuss it when we make the config files... in 
separate issue/project(perhaps)

for the formats,
about the SQLite, forget it... it's too large for our embedded environment, we 
are working in 64KB space (atleast for now)... and sqlite would need 1MB or 
more.

about the .ini - it would not be hard to make parser for it, but sections will 
need some more care... I would go for the ML format (see comment #4 for the 
link) it is easy parsable and have sections like the ini file. and dont forget 
the code is already _written_, and for such environment like ours.

about the .xml - writing a parser - i'm not going to spend time on this... i 
even wont think of writing one on the PC. About porting existing one - i see no 
point of doing it, it will be large (and will become a lost space in unneeded 
parser), even the XML format is not that easy to edit in text editor and only 
complicates the thing... Anyway, in the camera there are some XML routines, but 
i wont give them much hope.

so the options are tag-based binary and ML's text file.

about the partition of the CF, i don't see why would we need a new partition, 
unless when you want to format the CF from the camera, but even in separate 
partition you wont be save, the camera will rewrite the partition table.
we can store our files in "A:\400PLUS\" folder for example, but what is the 
point of that ?

Original comment by fired...@gmail.com on 1 May 2011 at 8:30

[GoogleCodeExporter]

@firedust, nobody is asking you to port or write an XML parser, or take on any 
other hefty file or database parsing dev effort.  Just the opposite actually.  
I only offer suggestions here to consider which was asked in Comment #8.  I 
don't know your development background at all, I only know mine and I have done 
all of these things, just not in an embedded environment.  Thanks for your 
comments on SQLite, my research and use did not show this limitation but it is 
not likely our best source anyway.

Yes, any separate PC GUI app for use with 400plus would be an entire separate 
project from 400plus - I just want all to keep this in mind as decisions are 
made for such things as file formats, that this additional project will likely 
happen.

I think you missed my point completely on the CF card partitioning and ability 
to have 400plus files in a hidden partition - let's just forget it about for 
now and possibly discuss it at a later time when it makes more sense.  There is 
plenty to do now on already identified Issues.

I propose you guys as the "devs" of this project just decide on your own about 
the file formats, you are the ones writing the code anyway.  I will take a 
backseat on this.  The "ML format" in comment #4 appears fine and offers 
everything 400plus needs and more - and the file is easily read and written to 
from just about any language in just about any O/S.

Original comment by mike.guf...@gmail.com on 2 May 2011 at 5:40

[GoogleCodeExporter]

i meant no offence in my comment, sorry if you saw it that way.
lets get back to the point...

i'm split on the both formats ...
text file will be good for the user, but as Edu suggested it could bring some 
troubles...
still not a big problem in my understanding.
binary file will be easier for parsing and strict ... then with the PC soft it 
still could be editable by user...

so i have no opinion on this right now ...
i say we should wait for it a little more ...

Original comment by fired...@gmail.com on 2 May 2011 at 6:35

[GoogleCodeExporter]

If it was done in a regular text file, divided in two parts with a big warning 
in the middle and a small warning at the top with general presets at the top in 
an easy to read format and all the other perhaps more troubling features in the 
bottom part I think it would be the best way to proceed because greater input 
in to advance the project and more user interaction could be facilitated by the 
sharing of the many possible preset sharing.
User could read, copy, paste, edit and share the presets they make with each 
other.
It would be a faster way to find out how some unknown functions work threw the 
brute force of the number of users.
Myself, I will be happy when someone figures out the mirror lock.
The more mistakes the more we learn.
As long as none of them are lethal lol
Is their any problems that a user could get into that couldn't be fixed?
Better write that one in bold or on another file.
Many hands make light work.
Together we could go threw all the details much faster.
I sure it would be fun for a lot of people to be included in the egg hunt.
That a could be an adventure to find an unknown function or play it more safe 
in expressing your self to other camera users with your personal perfect preset 
settings that they could easily add to their on camera or adapt their settings 
to with a quick copy and paste. I think it would make for a greater more 
diverse community with more sharing and easy access.

Original comment by rona...@nbnet.nb.ca on 3 May 2011 at 11:47

[GoogleCodeExporter]

I think you are completely missing the point about what these CONFIGURATION 
files are, and what can be expected from them...

Original comment by eduardo....@gmail.com on 3 May 2011 at 12:12

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Is that a good thing?
I think a lot of people maybe missing those points because they don't get to 
read it in an easy to access format. 
More people having more access to more points would be bonus points.

Original comment by rona...@nbnet.nb.ca on 3 May 2011 at 1:01

[GoogleCodeExporter]

Those files contain CONFIGURATION parameters, not FUNCTIONS.

All that there is in those files are the VALUES of the ISO, aperture, shutter 
speed, custom functions, scripts' settings, ... ; most of those VALUES can be 
set from the camera, most of the rest can be set from 400plus, and that's 
pretty much all.

Magic is not going to happen just because you trow some random VALUES there: we 
already know which PARAMETER controls the mirror lock-up, and what VALUES 
enable or disable it; what we do not know is which FUNCTION raises it.

Original comment by eduardo....@gmail.com on 3 May 2011 at 1:12

[GoogleCodeExporter]

My .02 again: I'm back to thinking binary tag-based for config files again... 
It's just the best idea IMHO and easy to implement.

Original comment by mike.guf...@gmail.com on 3 May 2011 at 1:34

[GoogleCodeExporter]

Could a table including the funclist text (is that the list of functions?) from 
the source be used in a text file to enable a virtual Kiss Army of users to 
find out what Function does what quickly threw brute force?

Perhaps some magic could come from the creative ways some of the more talented 
photographers style of setting up their camera and presets if they could share 
them more easily.

Original comment by rona...@nbnet.nb.ca on 3 May 2011 at 1:36

[GoogleCodeExporter]

lol, maybe it could be made like little a game script.
Hit button for a not so random function.(users part of list?)
Then user enters report on what camera did.
If they find anything interesting it would be like like winning a prize.

Original comment by rona...@nbnet.nb.ca on 3 May 2011 at 1:58

[GoogleCodeExporter]

Enough is enough...

Original comment by eduardo....@gmail.com on 3 May 2011 at 2:34

[GoogleCodeExporter]

@rona: Regarding your statement in comment #21.

"Perhaps some magic could come from the creative ways some of the more talented 
photographers style of setting up their camera and presets if they could share 
them more easily."

I agree, the sharing of camera config and preset values has great advantages 
for photographers to experiment and learn new shooting styles from others (and 
this works both ways).  Since the Presets are currently a work-in-progress at 
this point and are still maturing into their final stages - any input into how 
you (realistically) might like to see and use them is valuable information to 
the project.

Be sure and take a look at the related recent project forum entries at: 
http://chdk.setepontos.com/index.php?topic=3290.1020 
as well current Issues #64 and #65 here to see how they are progressing, and 
the challenges the devs will have in making them happen.

Original comment by mike.guf...@gmail.com on 4 May 2011 at 4:22

[GoogleCodeExporter]

as much as I would personally prefer the ascii/txt format for convenience, I 
would also be afraid that an accidental typo might brick my camera.

my preference, particularly assuming mike's offer to write a desktop interface, 
would be to use a binary format for the settings file.  this assumes of course 
that mike does a fantastic job of parsing the values saved back to the settings 
files!

now as mike's editor progresses it would be wonderful if THAT application is 
able to load and save plain text settings files for purposes of interchange.  
The idea here is that we would still be able to share settings (txt format), 
the actual format files read by the camera would be better protected (binary) 
and the heavy load of parsing the text files for having proper values would 
fall on Mike's application, and not so much on the camera.

Variation: can each setting be saved to a separate file, and have the name of 
the file be based on the setting name?  then people can just copy/share the 
binary settings files without any external editor needed.

does this make sense?

Original comment by pe...@theis.us on 16 May 2011 at 12:46

[GoogleCodeExporter]

I guess I should have read all threads first... please disregard the variation 
comment!

Original comment by pe...@theis.us on 16 May 2011 at 12:47

[GoogleCodeExporter]

As this project moves towards one of the next future stable featured builds - 
we will soon begin a new sister project which will offer editing and sharing of 
400plus settings, configurations, etc. on a PC.  Not quite there yet with how 
400plus is going to finalize its own storing of such values, so for now a PC 
app is still just an alpha proof-of-concept.  Any and all ideas for how you 
might like to see such a PC app work with 400plus data are welcomed and 
appreciated.

Original comment by mike.guf...@gmail.com on 16 May 2011 at 5:24

[GoogleCodeExporter]

I see the following as how I think it would work the best for the most.

Displayed in the app as a table it would be easy to see every thing. Maybe with 
preset names going up one side and the settings across another in a 2d array.

If each preset was saved in its own file it might make importing/exporting 
simpler.

If they were recorded in ASCII in a not too cryptic manor, people could share 
them in messages with out needing to Up nor Download. They could copy it into 
their text and the receiver could past them in to a new file.

But i am sure I will be able to work with any way found to be better.
Thank you for asking.

Original comment by rona...@nbnet.nb.ca on 16 May 2011 at 9:11

[GoogleCodeExporter]

>  I would also be afraid that an accidental typo might brick my camera.
if it is a text file, you will never brick your camera. the only problem will 
be that some setting would not be applied if not parsed correctly... this could 
lead to undesired result, but not brick/break your camera.

> If they were recorded in ASCII in a not too cryptic manor, people could share 
them in messages with out needing to Up nor Download.
if they are in ascii, there won't be crypto. 
and i do not see big advantage in sharing the presets with other ppl.
only for bug reporting purposes perhaps.

Original comment by fired...@gmail.com on 16 May 2011 at 11:21

[GoogleCodeExporter]

Sorry for not being more clear in the above, when I said "not too cryptic 
manor", I could have more accurately said," in a form that can be easily read 
my most camera users with out needing another reference as an aid." 

Original comment by rona...@nbnet.nb.ca on 17 May 2011 at 9:39

[GoogleCodeExporter]

If they were readable they would make informative tag lines on photos for 
people who want to talk about photography and become better photographers.
There are uncountable excellent configurations for this software, to able to 
communicate them with a quick "copy and paste" would save the most helpful of 
people several lines of typing each time they wish to say how it was set when 
the shot was perfected or just document their style of photography.
I wonder if an auto save of the settings could be inserted into the data on 
each photo?

Original comment by rona...@nbnet.nb.ca on 20 May 2011 at 9:15

[GoogleCodeExporter]

If it is possible to write them in to the picture files, could groups of their 
thumbnails be used for a selection menu?
Then you could have as many presets as you have pictures on camera.
The first few could be used as your base go to settings and the menu looking 
like the 9 thumbnails you see in the zoomed out jump to screen in playback.
That I imagine would be a lot of work to figure out but would look flashy.
A photo selected setting restore might be of use also while experimenting with 
many different settings in several pictures then returning to the best without 
needing to think twice while doing a quick review.
But I am not sure that I still wouldn't prefer the very well done, nice, clear 
and quickly read text menu. 
Another thing about having camera data in the picture files, if it is some 
where it can be easily read, people will see it and may discover they would 
like their cameras to have those settings. 

Original comment by rona...@nbnet.nb.ca on 21 May 2011 at 5:04

[GoogleCodeExporter]

so, since one year passed, shall we close this issue as WontDo, or we can do 
this with the .INI files we have already ?

Original comment by fired...@gmail.com on 5 May 2012 at 1:22

[GoogleCodeExporter]

We make configuration files obsolete on each release, and I still would like to 
fix that.
The INI format is acceptable for me, better than the current fixed format.

Original comment by eduardo....@gmail.com on 5 May 2012 at 9:35

[GoogleCodeExporter]

Yes, I think it will be a good solution to the problem.

Original comment by fired...@gmail.com on 5 May 2012 at 10:40

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by fired...@gmail.com on 7 Feb 2013 at 2:57

• Added labels: Milestone-M3