Issues with form charsets ( national letters in form and controls Caption field)

[KestasL]

ℹ Computer information

• VFP version: 9
• FoxBin2PRG version: last one
• Last FoxBin2PRG version w/o bug (if available):

📝 Provide detailed reproduction steps (if any)

1. Create a form and set caption with national letters example: 'ĄŠČŲŪŽĮĘĖąščųūžįęė' (there is Lithuanian letters).
2. Convert form to sc2,
3. Convert form back to scx.
4. Open form and see what happened,

the same issue with all controls where is caption (text data)

✔️ Expected result

Get the same form as before conversion

❌ Actual result

Caption letters not as in source document

📷 Screenshots

Original form: Form generated back from sc2:

[lscheffler]

@KestasL please zip the example and attach.

What codepage do you use?

[KestasL]

There is example files: form_charsets_issue.zip in folder original - original form form1.scx form1.sc2 converted text file folder sc2_to_scx - converted form from sc2 file

[lscheffler]

The problem seams to be FontCharSet, the property is not in the sc2_to_scx

[KestasL]

@lscheffler, maybe you have quick solution how to fix that? Until you made changes and update git repository?

[lscheffler]

Right now I do not see what the problem is. The property is restored by FoxBin2Prg, I see no major difference inside the SCX files - only the property is not active in the designer. There is something odd in the scx (vcx) structure I need to learn first.

[lscheffler]

@KestasL Sorry, the last message was a mistake, I looked at the wrong page of the properties form.
All works like a charm with me. For security I've downloaded the foxbin2prg.prg from this repo, run it against your sc2 and it runs without problem.
Please try again with https://github.com/fdbozzo/foxbin2prg/blob/master/foxbin2prg.prg

[KestasL]

@lscheffler I probably didn't explain clearly. The problem is not that something doesn't open. The problem is that the letters in the original version and the one already converted from sc2 are different. In the example, I wrote Lithuanian letters, and if e.g. the title is a Lithuanian word, so after it is converted to sc2 and restored to scx, the word is distorted:

[lscheffler]

I understand that you different captions. The question is why. So I compare the forms:
Your form with properties in directory sc2_to_scx:

and your form in original directory, and as I regenerate it:

I can not display the difference, since my comp does not have the eastern char sets.

Switching the FontCharset should fix the problem. (Please try) The Caption is fine (both forms keep the same ASCII), only wrong displayed. The question is, why, on your comp, the property FontCharset is not regenerated.
It exists in the sc2 file, and regenerates on mine comp, with the recent version of foxbin2prg.prg and with the one from this repo, so please try to regenerate with the original prg from this repository for testing.

You can try to compare the caption strings.

• Open both forms
• activate command window
• move mouse pointer over the first form
• in command window, type loA = SYS(1270)
• move mouse pointer over the second form
• in command window, type loB = SYS(1270)
• type ?loA.caption==loB.caption

[KestasL]

@lscheffler In my case it's not the same: In my case it's not the same: If you look at the form1.sct files (which I posted before, before and after the conversion) with a hex editor, you will also see the difference. In the file, the text is coded differently.

[lscheffler]

Open the scx's via use, and compare the fields. You will see that record 3, field Properties is different (aside uniqueid), on the sc2_to_scx FontCharset is missing Check caption, this is the plain text.

Have you tried to set FontCharset to Baltic on your comp for the sc2_to_scx form? What is the result?

Try to print the captions of both forms like

?0h+loA.caption
?0h+loB.caption

and compare the Hex values

[KestasL]

@lscheffler I don't think the problem is in the selected FontCharset. The entire form database is in a different encoding. Therefore, when translating texts from one encoding to another, discrepancies occur. You just need to save the coding somewhere and rebuild the form with the code again. Here are the encodings for the forms database:

[lscheffler]

You use "form database" in the sense of scx table?

Please

• Try to alter FontCharset
• Check the Hex values

And return the result.

What I see

I check the code page problem, anyway, as you see in the .2 files, there is no information about codepage stored, so for recent text files there is no possibility to create a binary source by a given CodePage.

Again, for me all the data looks the same, but I see the different codepages of the SCX's.
top is original, bottom is sc2_to_scx, as well as the forms in the form designer, as you see in https://github.com/fdbozzo/foxbin2prg/issues/96#issuecomment-1671549999. This all is from your zip unaltered.

Problems

We have 2 problem right now

1. for the future, keep the codepage.
   • since I know nothing about CodePage, I must please you to check your binary code for codepages, what files come with codepage? This is for PJX,VCX,SCX,FRX,LBX,MNX and DBC files, those all are just tables and could therefore have a codepage. The CodePage of a DBF is handled by FoxBin2Prg.
2. How to get your files to work in the right codepage, after we figured out how to store and restore.

[KestasL]

@lscheffler All file tables (PJX,VCX,SCX,FRX,LBX,MNX) in the original project (where not changed with foxbin2prg) have encoding 1257, after converting to text format and back, the encoding changes to 1252. There are no DBC files in our project, so I didn't check them, but I think it's the same there.

You use "form database" in the sense of scx table?

No, I just don't know much about how FoxPro works and I named the table as a database.

[lscheffler]

First question

There is still the problem with the not recreated FontCharset property. Have you tested the recreation with the recent foxbin2prg.prg? I must insist on this. I will not publish a new version if a bug like this exists.

Files, local code page

Thank you for looking it up.

-The codepage ?CPCURRENT() you use is set via config.fpw or is this your systems setting ?CPCURRENT(1)?

• Please check the CodePage of a new created table.

Your systeme(s)

Do you use different computers (or computer users) to recreate Bin2Text / Text2Bin with FoxBin2Prg? Do the computers use the same codepage? Or have you changed the computer?

• Please check the source computers codepage (in VFP ?CPCURRENT()) vs the target computers codepage.
• Try to create a table on command line and check CPDBF()

I learned a bit about the problem.

• A table created in VFP will by default use the active codepage.
• One can use a different codepage by issuing CREATE TABLE .. CODEPAGE=nCodePage
• FoxBin2Prg uses the default codepage

Conclusion:

• Your project was created (this says nothing about the computer using the project) with a computer / VFP using CodePage 1275. (That gives all the files the codepage 1275)
• This automatically created all tables - iow all binary VFP sources - to CodePage 1275
• The computer / VFP used the regenerate is set to CodePage 1252.
• This automatically regenerate anything to CodePage 1525
• The quick fix is setting your VFP to codepage 1275 and regenerate.
  • If you need help to set the codepage for VFP, please ask.
  • I think you must set the codepage for further development anyway.
• FoxBin2Prg needs to store this information for the files

Off topic

Please look up how the terms CURSOR, TABLE and DATABASE are used in VFP, to avoid confusion.

[KestasL]

@lscheffler Sorry, I don't have that much time to test the system and write answers. I thought you might be looking for ideas on how to quickly fix the problem. I will try to find out and fix the error in my version of foxbin2prg myself, or at least I will specify everywhere in my version to use the encoding 1257 (Lithuanian). P.S. my system encoding ?CPCURRENT() = 1257 ?CPCURRENT(1) = 1257 ?CPCURRENT(2) = 1252 if that helps you. Besides, it doesn't matter what encoding my texts are, you can try everything with some other national encoding (where not only English letters). The problem is that the conversion is happening, and ideally there shouldn't be. When and if I solve the problem myself, I will be write how I solved it.

[lscheffler]

Look, for me this is a side project I've inherited. I do my best - but I do it in my spare time.
On my comp, with the recent version off FoxBin2Prg from this repo as of this moment there is no problem, if codepage is set the right way. There is one regenerating on a different codepage, but this might be minor.
So to find the problem I either need your comp (hardware or remote - but you might have good reasons to not do that) or your assistance.

All I see is:

• FoxBin2prg will recreate to the CodePage set in the fpw config file, or to the default
  • this is tested on my comp and works
  • if your VFP is set to 1257 and does not create tables in 1257 I have no idea. Possibly there is a setting turning codepage off for the whole system, I have no idea. Either I compare with yours - or I have no chance.
• Your example is missing the FontCharset in the scx recreated
  • The property is number fine recreated with the recent version
  • Either your example is from a different sc2, altered or something else is wrong on your side, this is beyond my knowledge.

There is nothing more I can do on this issue until I get more information. I'm really sorry, but I will not start to learn each corner of CodePage to learn where the problem is. This is some magnitudes more work then some simple tests. :(

The recreate-codepage-on-a-different-machine problem will go to a separate issue, this will be reserved to the problems of your setup.

[lscheffler]

@KestasL There is probably a way to alter your files to the right codepage.

• you need the file HOME(1)+"tools\cpzero\cpzero.prg" (it might not compile there, so copy to a better place)
• this can alter the codepage of a given table (iow binary source)
• issue help cpzero.prgon command prompt for usage
• run for all your files

I consider this a kludge, but if it gets your code running, use this until the the base problem is solved.

Update:
Check help for topic How to: Specify the Code Page of a .dbf File

[KestasL]

@lscheffler, thank you for your efforts and I understand that this is just a hobby project. That's why I wrote that it will probably be faster for me to fix what is needed myself than to test what is wrong and describe it.

I partially solved the problem with localization (partially, because I solved only what is relevant for me). What I did: I save the codepage of each table (forms, libraries, menus, projects, etc.), while converting to text format I set the fields of the memo tables as binary. This was necessary to do that in order the data in the text file would also have the same encoding as the original ( I don't want if the code contains text, it will be distorted when tracking changes through git.). Later, when restoring files from text format to binary, (when creating tables), I set again the same codepage as it was in the original and place the data there. Since no data conversion takes place, we get analogue data as the original. I am attach a prg file, you can compare it with your version and see what was change. P.S. You'll still have a little trouble comparing my changes, because the foxbin2prg.prg file has a lot of Spanish comments, and my national encoding doesn't have Spanish letters, so the comments have changed a bit. But I think you will understand what was changed.

In addition, without these changes, not only the headers of the form components, but also the text in the code could have been distorted.

foxbin2prg.zip

[lscheffler]

I appreciate your help.
Only - there are 310 diffs in this file (mostly the comments).
Please:
There is a page showing how to contribute linked via README.md

Also, please make sure that if you are working on something that points toward the community, run VFP in CodePage 1252, or an editor that not touches codepage.

At least, would you mind to place a comment on each change? Just like:

*KestasL YYYYMMDD
*reason of change
code
*/KestasL YYYYMMDD

Logically any text will change, if the codepage of a table changes. This is expected. Just alter the codepage of any table and watch it happen. This is the sense of ancient codepage - tell the comp how to display a bunch of ASCII.

---

Aside of that, I still do not see WHY it does not regenerate. I simply set my VFP to your codepage and FoxBin2PRG will regenerate all stuff into this code page. I get the identical SCX. If this does not happen on your comp. There is a general problem on your side. You should figure out why. That's what my questions above are about. Until that, I don't think I will alter the code.

[KestasL]

@lscheffler I don't want to be a contributor to the project, I'm just sharing what I've done so you can take a look and if you think it's right, then update the project. I also don't think it's a good idea to write comments in code when using git. You can always write comments when committing updates and review them later with git. Similarly, it is a bad idea to write comments in Spanish (because I would much rather understand the code from English variable and function names than to translate comments with google translate), although English is also not my native language :).

Now to the point :): I fixed the error because I was not very good at describing the conversion of data tables from text format. And with 'WinMerge' I moved the comments from the original, so maybe there won't be so many changes now. Check it out, and good luck with your project development.

foxbin2prg.zip

Aside of that, I still do not see WHY it does not regenerate. I simply set my VFP to your codepage and FoxBin2PRG will regenerate all stuff into this code page. I get the identical SCX. If this does not happen on your comp. There is a general problem on your side. You should figure out why. That's what my questions above are about. Until that, I don't think I will alter the code.

The project that needs to be prepared for the work of several programmers (which needs to be supported) and on which I will have to work has many forms, reports (the number of forms alone is 2135) and codings are different (1251,1252,1257), probably it depended on the mood of the programmer :). Therefore, it is necessary that the coding is not converted to one format, but everything is restored as it was. In my opinion, this is the most correct.

[lscheffler]

@KestasL Thanks for the file. I will look it up.

Yes, I think the codepage should be kept and not depend on the computer. Your problem seems complex enough.

[lscheffler]

Looks fine so far - except the missing comments. :(

[lscheffler]

@KestasL I do not understand how and where do you store the CODEPAGE.
I see code to store the data plain ASCII, I see code to create tables with codepage, I see you reading the tables codepage.

What I do not see how or where the codepage is stored to the *.??2 files. Could you point me to the code doing this / give me an example text file?

[KestasL]

@lscheffler Code page is stored in .??2 files in header: `< FOXBIN2PRG: Version="1.19" SourceFile="alarm.scx" CPID="1257" /> (Solo para binarios VFP 9 / Only for VFP 9 binaries)` CPID="1257" there is codepage

[lscheffler]

@KestasL Ok, that was the impression I had.
Only the value is not written in the zip from above. Never mind, I can fix this.

[lscheffler]

Fixed version is available at https://github.com/lscheffler/foxbin2prg, no Thor update provided.

[lscheffler]

@KestasL Just to notify you. There is a bug in the version you posted. (Not your bug, it was there all the time) It fails recreating a MNX, the toModulo is not set to .NULL. in c_conversor_prg_a_mnx.convert before calling .createMenu .
A fixed version is available at https://github.com/lscheffler/foxbin2prg/releases/tag/v1.20.05

[KestasL]

Hi,

Since the project I'm working on
  doesn't have a menu (menus are created programmatically), I didn't
  notice this error. 

Thank you, I will check the new version
  (a little later).

Best regards,

  Kęstutis Laurinavičius

On 2023-09-03 16:49, Lutz Scheffler
  wrote:

  @KestasL
    Just to notify you. There is a bug in the version you posted.
    (Not your bug, it was there all the time) It fails recreating a
    MNX, the toModulo is not set to .NULL. in
    c_conversor_prg_a_mnx.convert before calling .createMenu .
    A fixed version is available at https://github.com/lscheffler/foxbin2prg/releases/tag/v1.20.05
  —
    Reply to this email directly, view it on GitHub, or unsubscribe.
    You are receiving this because you were mentioned.Message
      ID: ***@***.***>
  [

{ @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/fdbozzo/foxbin2prg/issues/96#issuecomment-1704311865", "url": "https://github.com/fdbozzo/foxbin2prg/issues/96#issuecomment-1704311865", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]