search

kikitte / GDAL4Android

Builds GDAL into an Android Archive(AAR) file
10 stars 2 forks source link

读写中文是乱码，以及写最大不能超过10个字符, 同时加载4个中文字崩溃！ #11

Closed lanzhu1993 closed 1 day ago

lanzhu1993 commented 3 days ago

我尝试用QGIS编辑创建Shp, filed属性字段用中文，并且设置编码方式为GBK 复现情况如下： 1、shapefile 编码格式是GBK，加载属性字段是中文乱码 2、shapefile字段长度是超过10个字符（中文超过3个），写和读都是乱码，同时被截取

kikitte commented 3 days ago

Hi,

  1. 貌似我尝试读取中文字段名会导致错误，由于gdal不支持gbk/gb2312编码，所以会将gb2312编码的字符串当成utf8解码处理导致异常，你指的加载属性字段是指读取字段名吗？ https://github.com/kikitte/GDAL4Android/blob/774185de7c64fe8e348a5fa4c331cc19e2b3928b/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L48-L52

如果只是读取字段值可参考 该函数实现。https://github.com/kikitte/GDAL4Android/blob/774185de7c64fe8e348a5fa4c331cc19e2b3928b/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L41

  1. shapfile数据规范要求如此，字段名的字节长度有限制。
lanzhu1993 commented 2 days ago

Hi,

  1. 貌似我尝试读取中文字段名会导致错误，由于gdal不支持gbk/gb2312编码，所以会将gb2312编码的字符串当成utf8解码处理导致异常，你指的加载属性字段是指读取字段名吗？ https://github.com/kikitte/GDAL4Android/blob/774185de7c64fe8e348a5fa4c331cc19e2b3928b/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L48-L52

如果只是读取字段值可参考 该函数实现。

https://github.com/kikitte/GDAL4Android/blob/774185de7c64fe8e348a5fa4c331cc19e2b3928b/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L41

  1. shapfile数据规范要求如此，字段名的字节长度有限制。

感谢博主提供这么好的库！希望这个库越来越好

第一个我在测试一下，目前我通过CSDN编译Linux gdal 3.0.1版本，通过gdal 读写中文，都是正常，同时读写shp也支持超过10个字符，具体链接如下：linux(阿里云服务器CentOS7)安装配置Java版GDAL环境，所以具体差异在哪里？

以及还有问题，找了很多资料，乱码的原因，在gdal 底层 GBK 编码格式转UTF默认回丢失，我不知道是不是这个原因引起的，具体链接如下： 【C#】C#中使用GDAL3（二）：Windows下读写Shape文件及超详细解决中文乱码问题](https://www.cnblogs.com/litou/p/15035790.html)

lanzhu1993 commented 2 days ago

测试步骤如下： 1.shapefile的字段名称如下图所示：

image

测试方法修修改如下： ` fun readChineseTestShp3(sb: StringBuilder, path: String): Unit { val chineseShpDataset = ogr.Open(path) val chineseShpLayer = chineseShpDataset.GetLayer(0) chineseShpLayer.ResetReading() val count = chineseShpLayer.GetLayerDefn().GetFieldCount() sb.append("count").append("=").append(count).append("\n")

// When the character length exceeds 3 Chinese characters, such as "样地类别" as an attribute field,
// obtaining the field name directly crashes
for (i in 0..<count) {
    val nameField = chineseShpLayer.GetLayerDefn().GetFieldDefn(i)
    val nameFieldStr = String(nameField.GetName().toByteArray(),Charset.forName("GBK"))
    sb.append(nameFieldStr).append("\n")
}

chineseShpDataset.delete()

} `

错误日志：

runtime.cc:691] at org.gdal.ogr.ogrJNI.FieldDefn_GetName(Native method) runtime.cc:691] at org.gdal.ogr.FieldDefn.GetName(FieldDefn.java:82) runtime.cc:691] at com.example.gdaltest.Asset_readerKt.readChineseTestShp3(asset_reader.kt:53) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2$1$1.invoke(MainActivity.kt:75) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2$1$1.invoke(MainActivity.kt:48) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.material3.SurfaceKt$Surface$1.invoke(Surface.kt:134) runtime.cc:691] at androidx.compose.material3.SurfaceKt$Surface$1.invoke(Surface.kt:115) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228) runtime.cc:691] at androidx.compose.material3.SurfaceKt.Surface-T9BRK9s(Surface.kt:112) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2$1.invoke(MainActivity.kt:46) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2$1.invoke(MainActivity.kt:44) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:248) runtime.cc:691] at androidx.compose.material3.TextKt.ProvideTextStyle(Text.kt:352) runtime.cc:691] at androidx.compose.material3.MaterialThemeKt$MaterialTheme$1.invoke(MaterialTheme.kt:72) runtime.cc:691] at androidx.compose.material3.MaterialThemeKt$MaterialTheme$1.invoke(MaterialTheme.kt:71) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) 2024-11-12 13:02:31.783 28106-28106 xample.gdaltest com.example.gdaltest A runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228) runtime.cc:691] at androidx.compose.material3.MaterialThemeKt.MaterialTheme(MaterialTheme.kt:64) runtime.cc:691] at com.example.gdaltest.ui.theme.ThemeKt.GDAL4AndroidTheme(Theme.kt:65) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2.invoke(MainActivity.kt:44) runtime.cc:691] at com.example.gdaltest.MainActivity$onCreate$2.invoke(MainActivity.kt:43) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.ui.platform.ComposeView.Content(ComposeView.android.kt:428) runtime.cc:691] at androidx.compose.ui.platform.AbstractComposeView$ensureCompositionCreated$1.invoke(ComposeView.android.kt:252) runtime.cc:691] at androidx.compose.ui.platform.AbstractComposeView$ensureCompositionCreated$1.invoke(ComposeView.android.kt:251) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228) runtime.cc:691] at androidx.compose.ui.platform.CompositionLocalsKt.ProvideCommonCompositionLocals(CompositionLocals.kt:186) runtime.cc:691] at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt$ProvideAndroidCompositionLocals$3.invoke(AndroidCompositionLocals.android.kt:119) runtime.cc:691] at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt$ProvideAndroidCompositionLocals$3.invoke(AndroidCompositionLocals.android.kt:118) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228) runtime.cc:691] at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt.ProvideAndroidCompositionLocals(AndroidCompositionLocals.android.kt:110) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1$1$2.invoke(Wrapper.android.kt:139) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1$1$2.invoke(Wrapper.android.kt:138) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:248) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(Wrapper.android.kt:138) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(Wrapper.android.kt:123) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:109) runtime.cc:691] at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35) runtime.cc:691] at androidx.compose.runtime.ActualJvm_jvmKt.invokeComposable(ActualJvm.jvm.kt:90) runtime.cc:691] at androidx.compose.runtime.ComposerImpl.doCompose(Composer.kt:3302) runtime.cc:691] at androidx.compose.runtime.ComposerImpl.composeContent$runtime_release(Composer.kt:3235) runtime.cc:691] at androidx.compose.runtime.CompositionImpl.composeContent(Composition.kt:725) runtime.cc:691] - locked <0x040aa992> (a java.lang.Object) runtime.cc:691] at androidx.compose.runtime.Recomposer.composeInitial$runtime_release(Recomposer.kt:1071) 2024-11-12 13:02:31.783 28106-28106 xample.gdaltest com.example.gdaltest A runtime.cc:691] at androidx.compose.runtime.CompositionImpl.composeInitial(Composition.kt:633) runtime.cc:691] at androidx.compose.runtime.CompositionImpl.setContent(Composition.kt:619) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:123) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:114) runtime.cc:691] at androidx.compose.ui.platform.AndroidComposeView.setOnViewTreeOwnersAvailable(AndroidComposeView.android.kt:1289) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition.setContent(Wrapper.android.kt:114) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition.onStateChanged(Wrapper.android.kt:164) runtime.cc:691] at androidx.lifecycle.LifecycleRegistry$ObserverWithState.dispatchEvent(LifecycleRegistry.kt:322) runtime.cc:691] at androidx.lifecycle.LifecycleRegistry.addObserver(LifecycleRegistry.kt:199) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:121) runtime.cc:691] at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:114) runtime.cc:691] at androidx.compose.ui.platform.AndroidComposeView.onAttachedToWindow(AndroidComposeView.android.kt:1364) runtime.cc:691] at android.view.View.dispatchAttachedToWindow(View.java:23244) runtime.cc:691] at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3722) runtime.cc:691] at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3729) runtime.cc:691] at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3729) runtime.cc:691] at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3729) runtime.cc:691] ... repeated 0 times runtime.cc:691] at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:3958) runtime.cc:691] at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:3345) runtime.cc:691] at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:11437) runtime.cc:691] at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1690) runtime.cc:691] at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1699) runtime.cc:691] at android.view.Choreographer.doCallbacks(Choreographer.java:1154) runtime.cc:691] at android.view.Choreographer.doFrame(Choreographer.java:1080) runtime.cc:691] at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1647) runtime.cc:691] at android.os.Handler.handleCallback(Handler.java:958) runtime.cc:691] at android.os.Handler.dispatchMessage(Handler.java:99) runtime.cc:691] at android.os.Looper.loopOnce(Looper.java:230) runtime.cc:691] at android.os.Looper.loop(Looper.java:319) runtime.cc:691] at android.app.ActivityThread.main(ActivityThread.java:9063) runtime.cc:691] at java.lang.reflect.Method.invoke(Native method) runtime.cc:691] at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:588) runtime.cc:691] at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103) runtime.cc:691]

lanzhu1993 commented 2 days ago

[libprotobuf ERROR external/protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'Tombstone.abort_message' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Build fingerprint: 'samsung/e1qzcx/e1q:14/UP1A.231005.007/S9210ZCS3AXI1:user/release-keys' 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Revision: '13' 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A ABI: 'arm64' 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Processor: '7' 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Timestamp: 2024-11-12 13:02:31.845408599+0800 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Process uptime: 2s 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Cmdline: com.example.gdaltest 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A pid: 28106, tid: 28106, name: xample.gdaltest >>> com.example.gdaltest <<< 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A uid: 10499 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE) 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY) 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr -------- 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A Abort message: 'JNI DETECTED ERROR IN APPLICATION: input is not valid Modified UTF-8: illegal continuation byte 0xd8 string: '??' input: '0xcf <0xd8>' in call to NewStringUTF from java.lang.String org.gdal.ogr.ogrJNI.FieldDefn_GetName(long, org.gdal.ogr.FieldDefn)' 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x0 0000000000000000 x1 0000000000006dca x2 0000000000000006 x3 0000007ff1039c90 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x4 5151441f43445342 x5 5151441f43445342 x6 5151441f43445342 x7 7f7f7f7f7f7f7f7f 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x8 00000000000000f0 x9 0000007f2b476278 x10 0000000000000001 x11 0000007f2b4beb60 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x12 000000000000c5a0 x13 0000000000000000 x14 0000007ff1038a90 x15 00000bb25d7e6fe7 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x16 0000007f2b527d18 x17 0000007f2b502680 x18 0000007f30ecc000 x19 0000000000006dca 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x20 0000000000006dca x21 00000000ffffffff x22 0000000000000002 x23 0000007c11bb7ba7 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x24 0000000000000002 x25 0000000000000001 x26 0000000000000000 x27 0000007c1272a000 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A x28 b400007cd30608a0 x29 0000007ff1039d10 2024-11-12 13:02:32.009 28155-28155 DEBUG pid-28155 A lr 0000007f2b4af704 sp 0000007ff1039c70 pc 0000007f2b4af730 pst 0000000000001000

kikitte commented 2 days ago

应当解决了。

GDAL输入输出涉及字符串的都会使用utf-8编码，所以碰到shapefile等使用其他文本编码格式(如gb2312)的文件格式时会将其转换为utf-8编码，字符串编码转换工作在gdal内部依赖iconv实现。 原有编译的aar没有iconv依赖，所以导致读取gb2312的shapefile时出错。 现在加了，应该没有问题了，包括读取中文字段名、中文属性值，也不需要额外在java代码进行编码转换工作。 https://github.com/kikitte/GDAL4Android/blob/6ea9b49229846690c7b908fd34e2c330846f6311/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L41-L68

至于字段名长度，按照网上的资料，shapefile格式支持的字段名长度最大为10，因此gb2312编码下最多支持5个中文字符。

lanzhu1993 commented 2 days ago

应当解决了。

GDAL输入输出涉及字符串的都会使用utf-8编码，所以碰到shapefile等使用其他文本编码格式(如gb2312)的文件格式时会将其转换为utf-8编码，字符串编码转换工作在gdal内部依赖iconv实现。 原有编译的aar没有iconv依赖，所以导致读取gb2312的shapefile时出错。 现在加了，应该没有问题了，包括读取中文字段名、中文属性值，也不需要额外在java代码进行编码转换工作。

https://github.com/kikitte/GDAL4Android/blob/6ea9b49229846690c7b908fd34e2c330846f6311/gdaltest/src/main/java/com/example/gdaltest/asset_reader.kt#L41-L68

至于字段名长度，按照网上的资料，shapefile格式支持的字段名长度最大为10，因此gb2312编码下最多支持5个中文字符。

可否加个WeChat, 我的邮箱1113799552@qq.com,你把你的V发我，我希望能加入你的项目，提一些PR，最近在学习gdal

lanzhu1993 commented 1 day ago

Hi, 我测试啦，目前中文适配读取的时候已经Work啦，字段长度限制，我查了资料，在3.0以上高版本目前可以支持自动扩展， 博主可以研究一下吗？ 具体链接如下： 关于SHP文件字段大小 截图如下：

image
kikitte commented 1 day ago

Hi, 我测试啦，目前中文适配读取的时候已经Work啦，字段长度限制，我查了资料，在3.0以上高版本目前可以支持自动扩展， 博主可以研究一下吗？ 具体链接如下： 关于SHP文件字段大小 截图如下： image

我指的是字段名长度，不是字段宽度。另外目前的gdal版本是3.7，你指的3.0的功能应该是具备的。