search

utelle / SQLite3MultipleCiphers

SQLite3 encryption extension with support for multiple ciphers
https://utelle.github.io/SQLite3MultipleCiphers/
MIT License
420 stars 77 forks source link

Difficulties compiling with ICU support after updating #173

Closed Qriist closed 2 months ago

Qriist commented 2 months ago

(This is definitely human error on my end.)

I noticed that I hadn't updated my Multiple Ciphers build (with the sqlar functions enabled) in a while, so I did that. No issues creating the DLL. I then noticed my ICU version was a couple years old so I tried to update (v71 -> v75).

It wasn't working so I tried to revert back to the old ICU version. However, during my tinkering I broke something to the point where MC won't build with ICU at all (though the non-ICU version does).

Would you please help me configure whatever I need to reenable building with ICU?

most recent log

Build started at 2:19 PM...
1>------ Build started: Project: sqlite3mc_libicu, Configuration: Release Win64 x64 ------
2>------ Build started: Project: sqlite3mc_lib, Configuration: Release Win64 x64 ------
3>------ Build started: Project: sqlite3mc_dllicu, Configuration: Release Win64 x64 ------
4>------ Build started: Project: sqlite3mc_dll, Configuration: Release Win64 x64 ------
1>sqlite3mc.c
4>sqlite3mc.c
1>C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\src\sqlite3patched.c(217319,24): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
1>(compiling source file '../src/sqlite3mc.c')
1>C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\src\sqlite3patched.c(217375,14): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
1>(compiling source file '../src/sqlite3mc.c')
3>sqlite3mc.c
2>sqlite3mc.c
3>C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\src\sqlite3patched.c(217319,24): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
3>(compiling source file '../src/sqlite3mc.c')
3>C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\src\sqlite3patched.c(217375,14): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
3>(compiling source file '../src/sqlite3mc.c')
4>   Creating library ..\bin\vc17\dll\release\sqlite3mc_x64.lib and object ..\bin\vc17\dll\release\sqlite3mc_x64.exp
1>sqlite3mc_vc17_libicu.vcxproj -> C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\bin\vc17\lib\release\sqlite3mc_icu_x64.lib
1>Done building project "sqlite3mc_vc17_libicu.vcxproj".
5>------ Build started: Project: sqlite3mc_shellicu, Configuration: Release Win64 x64 ------
5>shell.c
4>sqlite3mc_vc17_dll.vcxproj -> C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\bin\vc17\dll\release\sqlite3mc_x64.dll
5>   Creating library ..\bin\vc17\lib\release\sqlite3mc_shellicu_x64.lib and object ..\bin\vc17\lib\release\sqlite3mc_shellicu_x64.exp
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_errorName_75 referenced in function icuFunctionError
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_isspace_75 referenced in function icuNext
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_foldCase_75 referenced in function icuOpen
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol uregex_open_75 referenced in function icuRegexpFunc
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol uregex_close_75 referenced in function icuRegexpDelete
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol uregex_setText_75 referenced in function icuRegexpFunc
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol uregex_matches_75 referenced in function icuRegexpFunc
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_strToUpper_75 referenced in function icuCaseFunc16
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_strToLower_75 referenced in function icuCaseFunc16
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol u_strToUTF8_75 referenced in function icuNext
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ucol_open_75 referenced in function icuLoadCollation
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ucol_close_75 referenced in function icuLoadCollation
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ucol_strcoll_75 referenced in function icuCollationColl
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ucol_setStrength_75 referenced in function icuLoadCollation
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ubrk_open_75 referenced in function icuOpen
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ubrk_close_75 referenced in function icuClose
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ubrk_current_75 referenced in function icuNext
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ubrk_next_75 referenced in function icuNext
5>sqlite3mc_icu_x64.lib(sqlite3mc.obj) : error LNK2019: unresolved external symbol ubrk_first_75 referenced in function icuOpen
5>..\bin\vc17\lib\release\sqlite3mc_shellicu_x64.exe : fatal error LNK1120: 19 unresolved externals
5>Done building project "sqlite3mc_vc17_shellicu.vcxproj" -- FAILED.
2>sqlite3mc_vc17_lib.vcxproj -> C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\bin\vc17\lib\release\sqlite3mc_x64.lib
3>   Creating library ..\bin\vc17\dll\release\sqlite3mc_icu_x64.lib and object ..\bin\vc17\dll\release\sqlite3mc_icu_x64.exp
6>------ Build started: Project: sqlite3mc_shell, Configuration: Release Win64 x64 ------
6>shell.c
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_errorName_75 referenced in function icuFunctionError
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_isspace_75 referenced in function icuNext
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_foldCase_75 referenced in function icuOpen
3>sqlite3mc.obj : error LNK2019: unresolved external symbol uregex_open_75 referenced in function icuRegexpFunc
3>sqlite3mc.obj : error LNK2019: unresolved external symbol uregex_close_75 referenced in function icuRegexpDelete
3>sqlite3mc.obj : error LNK2019: unresolved external symbol uregex_setText_75 referenced in function icuRegexpFunc
3>sqlite3mc.obj : error LNK2019: unresolved external symbol uregex_matches_75 referenced in function icuRegexpFunc
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_strToUpper_75 referenced in function icuCaseFunc16
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_strToLower_75 referenced in function icuCaseFunc16
3>sqlite3mc.obj : error LNK2019: unresolved external symbol u_strToUTF8_75 referenced in function icuNext
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ucol_open_75 referenced in function icuLoadCollation
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ucol_close_75 referenced in function icuLoadCollation
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ucol_strcoll_75 referenced in function icuCollationColl
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ucol_setStrength_75 referenced in function icuLoadCollation
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ubrk_open_75 referenced in function icuOpen
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ubrk_close_75 referenced in function icuClose
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ubrk_current_75 referenced in function icuNext
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ubrk_next_75 referenced in function icuNext
3>sqlite3mc.obj : error LNK2019: unresolved external symbol ubrk_first_75 referenced in function icuOpen
3>..\bin\vc17\dll\release\sqlite3mc_icu_x64.dll : fatal error LNK1120: 19 unresolved externals
3>Done building project "sqlite3mc_vc17_dllicu.vcxproj" -- FAILED.
6>   Creating library ..\bin\vc17\lib\release\sqlite3mc_shell_x64.lib and object ..\bin\vc17\lib\release\sqlite3mc_shell_x64.exp
6>sqlite3mc_vc17_shell.vcxproj -> C:\Users\Qriist\source\repos\SQLite3MultipleCiphers\bin\vc17\lib\release\sqlite3mc_shell_x64.exe
========== Build: 4 succeeded, 2 failed, 0 up-to-date, 0 skipped ==========
========== Build completed at 2:20 PM and took 44.936 seconds ==========
utelle commented 2 months ago

I noticed that I hadn't updated my Multiple Ciphers build (with the sqlar functions enabled) in a while, so I did that. No issues creating the DLL. I then noticed my ICU version was a couple years old so I tried to update (v71 -> v75).

The latest release of SQLite3 Multiple Ciphers was built against an older ICU version, namely 74.2, but I just checked that the build also succeeds when using ICU 75.1.

It wasn't working so I tried to revert back to the old ICU version. However, during my tinkering I broke something to the point where MC won't build with ICU at all (though the non-ICU version does).

Would you please help me configure whatever I need to reenable building with ICU?

The Visual C++ build files use the environment variable LIBICU_PATH to point to the root directory of your ICU installation (where subfolders include and lib resp lib64 reside). The compiler uses it to find the ICU header files, and the linker uses it fo find the ICU lib files. Maybe you changed the name of the environment variable - but not in all places.

AFAICT from your log only the linker complains about not finding the required ICU entry points, while the compiler finds the header files. Therefore I suspect that the library path for the linker is somehow broken. Maybe it still points to the previous ICU version. Or you copied the library files of the wrong version to the lib resp lib64 folder.

Qriist commented 2 months ago

The latest release of SQLite3 Multiple Ciphers was built against an older ICU version, namely 74.2, but I just checked that the build also succeeds when using ICU 75.1.

Oops, didn't realize I had eclipsed your release. My bad! Thank you for confirming the most recent ICU works.

The Visual C++ build files use the environment variable LIBICU_PATH to point to the root directory of your ICU installation (where subfolders include and lib resp lib64 reside).

Thankfully, that is what I had done.

Therefore I suspect that the library path for the linker is somehow broken.

Well, that got me thinking.

========== Build: 6 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
========== Build completed at 4:29 PM and took 41.622 seconds ==========

Apparently Visual Studio was in dire need of a PC reboot. That's it, that fixed the compiler issue. Maybe something got locked from a recent compiler update that I didn't notice. ¯_(ツ)_/¯

HOWEVER another oddity cropped up: I have confirmed that I am targeting v75... image image

...but the ICU dll seems to still request v71. image v75 never gets requested. This persists even after a rebuild following removing the icu-precompiled/icu-71.1-vs2019 sibling directory (which obviously shouldn't matter but I did anyways just to be sure).

Aside from the Windows environment variable, is there another location where I need to set LIBICU_PATH?

utelle commented 2 months ago

Therefore I suspect that the library path for the linker is somehow broken.

Well, that got me thinking.

========== Build: 6 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
========== Build completed at 4:29 PM and took 41.622 seconds ==========

Good news.

Apparently Visual Studio was in dire need of a PC reboot. That's it, that fixed the compiler issue. Maybe something got locked from a recent compiler update that I didn't notice. ¯(ツ)/¯

Well, especially if environment variables are involved it is often necessary to restart at least Visual Studio or even to reboot the computer, so that changes take effect.

HOWEVER another oddity cropped up: I have confirmed that I am targeting v75... ...but the ICU dll seems to still request v71.

This is a bit strange, because it indicates that the executable you are invoking was not rebuilt properly. How do you invoke the executable?

This persists even after a rebuild following removing the icu-precompiled/icu-71.1-vs2019 sibling directory (which obviously shouldn't matter but I did anyways just to be sure).

Maybe you'll have to clear Visual Studio's cache (via Build/Clean Solution from the VS menu) to remove remains from earlier builds.

Aside from the Windows environment variable, is there another location where I need to set LIBICU_PATH?

The VS solution has 3 projects, sqlite3mc_dllicu, sqlite3mc_libicu, and sqlite3mc_shellicu, which reference LIBICU_PATH in the C++ and/or Linker section. Make sure all references are properly set.

Qriist commented 2 months ago

tl;dr: finally properly compiled! :D

Well, especially if environment variables are involved it is often necessary to restart at least Visual Studio or even to reboot the computer, so that changes take effect.

Understandable!

This is a bit strange, because it indicates that the executable you are invoking was not rebuilt properly. How do you invoke the executable?

My preferred SQLite GUI is SQLiteStudio. It ships with a vanilla SQLite3.dll that I drop-replaced with the MC+ICU+SQLAR binary. This worked flawlessly with the previous MC build that I was updating from, with the added advantage of the GUI having a robust error handler to give me visual feedback. SQLiteStudio auto-loads SQLite3.dll when launched, plus the ICU dlls if given that MC variant. SQLiteStudio does not use the ICU dlls, making this an ideal test scenario to see if I've built MC correctly. That's how I was able to determine MC was still building against v71.

(My most common use-case for MC is any number of headless scripts and libraries I've created. However, since those are deliberately designed to give little-to-no feedback they don't work well for debugging of this type.)

Maybe you'll have to clear Visual Studio's cache (via Build/Clean Solution from the VS menu) to remove remains from earlier builds.

The VS solution has 3 projects, sqlite3mc_dllicu, sqlite3mc_libicu, and sqlite3mc_shellicu, which reference LIBICU_PATH in the C++ and/or Linker section. Make sure all references are properly set.

I was finally able to properly build against v75 by basically going scorched earth on reseting everything.

  1. Deleting and re-cloning MC
  2. Setting the 4 required preprocessors on the ICU builds
  3. Cleaning the solution
  4. Closing the solution
  5. Closing Visual Studio
  6. Manually replacing the v75 files (again)
  7. Confirming my LIBICU_PATH was set correctly in Windows
  8. Rebooting the PC
  9. Opening VS and building (Notably, I did not set LIBICU_PATH anywhere within the project itself)

I had taken all of these actions sorta piecemeal over the past day, but not all together in this order. My best guess is that something somewhere got bogged down in cache hell, probably related to the first linker issue I had. My suffering has come to an end. XD

ANYWAYS, thank you very much for taking the time to help troubleshoot my idiotic issues. You've been an amazing help and I'm grateful. :)

Qriist commented 2 months ago

Since it was such an obnoxious ordeal for me, here's the latest MC+ICU+SQLAR for anyone else who might get caught in build purgatory. (Also includes non-ICU with SQLAR enabled.) SQLite (MC v1.8.7 + ICU v75 + SQLAR enabled).zip

utelle commented 2 months ago

tl;dr: finally properly compiled! :D

Glad to hear that. 😄

My preferred SQLite GUI is SQLiteStudio. It ships with a vanilla SQLite3.dll that I drop-replaced with the MC+ICU+SQLAR binary.

AFAIK SQLiteStudio comes with a SQLite3MC plugin - but it is somewhat dated and does not support ICU.

This worked flawlessly with the previous MC build

Yes, SQLite3MC can be used as a drop-in replacement for SQLite.

I was finally able to properly build against v75 by basically going scorched earth on reseting everything.

  1. Deleting and re-cloning MC
  2. Setting the 4 required preprocessors on the ICU builds
  3. Cleaning the solution
  4. Closing the solution
  5. Closing Visual Studio
  6. Manually replacing the v75 files (again)
  7. Confirming my LIBICU_PATH was set correctly in Windows
  8. Rebooting the PC
  9. Opening VS and building (Notably, I did not set LIBICU_PATH anywhere within the project itself)

Maybe not all steps would have been necessary, but to be on the safe side it is a reasonable approach.

Quite often it is enough to delete the .vs subfolder to force Visual Studio to rebuild a project really from scratch.

I had taken all of these actions sorta piecemeal over the past day, but not all together in this order. My best guess is that something somewhere got bogged down in cache hell, probably related to the first linker issue I had.

In the past I have experienced similar issues with Visual Studio myself.

My suffering has come to an end. XD

Great. 🎉

ANYWAYS, thank you very much for taking the time to help troubleshoot my idiotic issues. You've been an amazing help and I'm grateful. :)

You are welcome.