jihuan1203 commented 9 months ago

您好，我制作了一个场景文字识别数据集，标注格式类似Icdar2015，用论文给出的模型进行了评估： {"precision": 0.8825484764542936, "recall": 0.7669715936446798, "hmean": 0.8207109737248841, "AP": 0}

无法计算AP值，我修改了dataset/icdar15/Evaluation_Protocol/script.py文件中 'CONFIDENCES':True,参数的值，改为True，报错：b'Line in sample not valid. Sample: 100 Line: 415,83,510,79,511,111,416,115 Error: Format incorrect. Should be: x1,y1,x2,y2,x3,y3,x4,y4,confidence' 我想知道confidence 需要怎么得到？

jihuan1203 commented 9 months ago

以下附上gt_img_1.txt部分内容、res_img_1.txt部分内容： gt_img_1.txt: 556,222,627,222,627,245,556,245,BBC 164,312,368,312,368,352,164,352,writeback 418,312,623,312,623,358,418,358,description 256,414,256,388,314,388,314,414,Read 317,414,317,391,357,391,357,414,the 545,388,625,388,625,414,545,414,market 360,388,490,388,490,417,360,417,description 228,424,270,424,270,447,228,447,and 363,447,363,424,400,424,400,447,the 403,426,521,419,522,447,404,454,questions 277,427,357,427,357,447,277,447,answer res_img_1.txt: 550,246,550,218,629,218,629,246 658,249,658,219,715,219,715,249 162,356,162,305,618,305,618,356 189,417,190,381,629,387,628,423 222,422,522,420,523,450,223,452 227,463,609,459,610,488,228,492 225,499,388,497,389,526,226,528 223,537,487,534,488,564,224,567 234,572,476,570,477,601,235,602

GXYM commented 9 months ago

您好，我制作了一个场景文字识别数据集，标注格式类似Icdar2015，用论文给出的模型进行了评估： {"precision": 0.8825484764542936, "recall": 0.7669715936446798, "hmean": 0.8207109737248841, "AP": 0}

无法计算AP值，我修改了dataset/icdar15/Evaluation_Protocol/script.py文件中 'CONFIDENCES':True,参数的值，改为True，报错：b'Line in sample not valid. Sample: 100 Line: 415,83,510,79,511,111,416,115 Error: Format incorrect. Should be: x1,y1,x2,y2,x3,y3,x4,y4,confidence' 我想知道confidence 需要怎么得到？

你算AP干啥，很少有论文报告AP，你可以计算一些文本示例区域内的平均置信度，当做confidence，也可以直接写1.0就可以

jihuan1203 commented 9 months ago

我想知道置信度的原因是，我用给定不同模型测试自制数据集，准确度召回率等都是一样的，甚至用TD500测试集跑结果都一位不差，想知道我是哪里出了问题。

---原始邮件--- 发件人: @.> 发送时间: 2023年11月28日(周二) 晚上8:35 收件人: @.>; 抄送: @.**@.>; 主题: Re: [GXYM/TextBPN-Plus-Plus] 仿Icdar2015数据集评估报错 (Issue #30)

您好，我制作了一个场景文字识别数据集，标注格式类似Icdar2015，用论文给出的模型进行了评估： {"precision": 0.8825484764542936, "recall": 0.7669715936446798, "hmean": 0.8207109737248841, "AP": 0}

无法计算AP值，我修改了dataset/icdar15/Evaluation_Protocol/script.py文件中 'CONFIDENCES':True,参数的值，改为True，报错：b'Line in sample not valid. Sample: 100 Line: 415,83,510,79,511,111,416,115 Error: Format incorrect. Should be: x1,y1,x2,y2,x3,y3,x4,y4,confidence' 我想知道confidence 需要怎么得到？

你算AP干啥，很少有论文报告AP，你可以计算一些文本示例区域内的平均置信度，当做confidence，也可以直接写1.0就可以

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

jihuan1203 commented 9 months ago

该问题解决，评估Icdar15数据集时，代码不会自动删除已有的submit.zip、gt.zip，需要手动删除原有的，将预测值压缩为新的submit.zip，放入dataset/icdar15中

AXNing commented 9 months ago

你好，我有一些问题。我使用我自己的数据集进行测试，使用仿TD500的测试文件，发现TD的gt中有angle，这个angle是怎么使用Xmax，Ymax，Xmin，Ymin来计算得到的呢。谢谢 @GXYM

jihuan1203 commented 9 months ago

我直接使用了Icdar15测试文件测试的TD500数据集

AXNing commented 9 months ago

我直接使用了Icdar15测试文件测试的TD500数据集他们的gt不一样，我这一直报错，gt压缩文件里面格式不对，我就按照TD的gt去改，但是angle不会计算

AXNing commented 9 months ago

我直接使用了Icdar15测试文件测试的TD500数据集

你自己制作的数据集是只打了四个点吗，我的点和框比较多，怎么去拟合icdar15的格式

jihuan1203 commented 9 months ago

我刚刚评估TD500分别用Idcar15脚本和TD500脚本都遇到问题，等会研究研究，晚上再给你答复。我的数据集标注文件是四点标注，举例： 556,222,627,222,627,245,556,245,BBC 164,312,368,312,368,352,164,352,writeback 418,312,623,312,623,358,418,358,description 256,414,256,388,314,388,314,414,Read 317,414,317,391,357,391,357,414,the 545,388,625,388,625,414,545,414,market 360,388,490,388,490,417,360,417,description

jihuan1203 commented 9 months ago

我使用TD500测试脚本TextBPN_deformable_resnet50_480.pth测试TD500，测试完之后，手动压缩为submit.zip gt.zip，放入dataset/TD500，运行script.py，得到结果： 0%| | 0/200 [00:00<?, ?it/s] 100%|██████████| 200/200 [00:00<00:00, 2355.64it/s] Calculated: {"precision": 0.008678881388621022, "recall": 0.015463917525773196, "hmean": 0.01111797405806053, "AP": 0} 我认为结果存在问题，目前尚未找到。 res_img_1.txt: 649,246,746,383,0.028563734271637657 639,674,671,720,0.0 666,686,758,728,0.1275095480885205 gt_img_1.txt: 911,1016,1544,1156,0.005137,None 12,776,1522,1005,0.045299,None

GXYM commented 9 months ago

我使用TD500测试脚本TextBPN_deformable_resnet50_480.pth测试TD500，测试完之后，手动压缩为submit.zip gt.zip，放入dataset/TD500，运行script.py，得到结果： 0%| | 0/200 [00:00<?, ?it/s] 100%|██████████| 200/200 [00:00<00:00, 2355.64it/s] Calculated: {"precision": 0.008678881388621022, "recall": 0.015463917525773196, "hmean": 0.01111797405806053, "AP": 0} 我认为结果存在问题，目前尚未找到。 res_img_1.txt: 649,246,746,383,0.028563734271637657 639,674,671,720,0.0 666,686,758,728,0.1275095480885205 gt_img_1.txt: 911,1016,1544,1156,0.005137,None 12,776,1522,1005,0.045299,None

为什么需要手动压缩，这里有自动帮你压缩评测的脚本：https://github.com/GXYM/TextBPN-Plus-Plus/blob/main/dataset/TD500/eval.sh 这个脚本如果报错，有一种可能是写文件还没写完，就执行压缩命令，可以适当添加一个sleep来消除这个错误。 ··· cd dataset/TD500 rm submit.zip cp $1/.txt submit cd submit/;zip -r submit.zip &> ../log.txt ; mv submit.zip ../; cd ../ rm log.txt python Evaluation_Protocol/script.py -g=gt.zip -s=submit.zip ··· TD500的评测结果要正常，先保证CV2的版本不要太高

GXYM commented 9 months ago

我使用TD500测试脚本TextBPN_deformable_resnet50_480.pth测试TD500，测试完之后，手动压缩为submit.zip gt.zip，放入dataset/TD500，运行script.py，得到结果： 0%| | 0/200 [00:00<?, ?it/s] 100%|██████████| 200/200 [00:00<00:00, 2355.64it/s] Calculated: {"precision": 0.008678881388621022, "recall": 0.015463917525773196, "hmean": 0.01111797405806053, "AP": 0} 我认为结果存在问题，目前尚未找到。 res_img_1.txt: 649,246,746,383,0.028563734271637657 639,674,671,720,0.0 666,686,758,728,0.1275095480885205 gt_img_1.txt: 911,1016,1544,1156,0.005137,None 12,776,1522,1005,0.045299,None

为什么需要手动压缩，这里有自动帮你压缩评测的脚本：https://github.com/GXYM/TextBPN-Plus-Plus/blob/main/dataset/TD500/eval.sh 这个脚本如果报错，有一种可能是写文件还没写完，就执行压缩命令，可以适当添加一个sleep来消除这个错误。 ··· cd dataset/TD500 rm submit.zip cp $1/.txt submit cd submit/;zip -r submit.zip &> ../log.txt ; mv submit.zip ../; cd ../ rm log.txt python Evaluation_Protocol/script.py -g=gt.zip -s=submit.zip ··· TD500的评测结果要正常，先保证CV2的版本不要太高

另外你用了DCN版本的模型，记得要正确编译DCN相关的文件，尤其是GPU的型号，不能在2080上编译，去3090上测试

  # make sure that the PATH of CUDA in setup.py is set properly with the right environment
  # Set the GPU you need in setup.py, if If you have different GPUs in your machine.

  cd network/backbone/assets/dcn
  sh Makefile.sh

  # setup.py 
  import os
  PATH ="{}:{}".format(os.environ['PATH'], "/opt/cuda/bin")
  # os.environ['CUDA_VISIBLE_DEVICES'] = "1"
  os.environ['PATH'] = PATH
  from setuptools import setup
  from torch.utils.cpp_extension import BuildExtension, CUDAExtension

jihuan1203 commented 9 months ago

感谢，关于DCN的建议真的帮我大忙。压缩问题，也是我困惑的点，我明天试试加上sleep试试。

---原始邮件--- 发件人: @.> 发送时间: 2023年12月1日(周五) 晚上10:01 收件人: @.>; 抄送: @.>;"State @.>; 主题: Re: [GXYM/TextBPN-Plus-Plus] 仿Icdar2015数据集评估报错 (Issue #30)

我使用TD500测试脚本TextBPN_deformable_resnet50_480.pth测试TD500，测试完之后，手动压缩为submit.zip gt.zip，放入dataset/TD500，运行script.py，得到结果： 0%| | 0/200 [00:00<?, ?it/s] 100%|██████████| 200/200 [00:00<00:00, 2355.64it/s] Calculated: {"precision": 0.008678881388621022, "recall": 0.015463917525773196, "hmean": 0.01111797405806053, "AP": 0} 我认为结果存在问题，目前尚未找到。 res_img_1.txt: 649,246,746,383,0.028563734271637657 639,674,671,720,0.0 666,686,758,728,0.1275095480885205 gt_img_1.txt: 911,1016,1544,1156,0.005137,None 12,776,1522,1005,0.045299,None

为什么需要手动压缩，这里有自动帮你压缩评测的脚本：https://github.com/GXYM/TextBPN-Plus-Plus/blob/main/dataset/TD500/eval.sh 这个脚本如果报错，有一种可能是写文件还没写完，就执行压缩命令，可以适当添加一个sleep来消除这个错误。 ··· cd dataset/TD500 rm submit.zip cp $1/.txt submit cd submit/;zip -r submit.zip &> ../log.txt ; mv submit.zip ../; cd ../ rm log.txt python Evaluation_Protocol/script.py -g=gt.zip -s=submit.zip ··· TD500的评测结果要正常，先保证CV2的版本不要太高

另外你用了DCN版本的模型，记得要正确编译DCN相关的文件，尤其是GPU的型号，不能在2080上编译，去3090上测试

make sure that the PATH of CUDA in setup.py is set properly with the right environment # Set the GPU you need in setup.py, if If you have different GPUs in your machine. cd network/backbone/assets/dcn sh Makefile.sh # setup.py import os PATH ="{}:{}".format(os.environ['PATH'], "/opt/cuda/bin") # os.environ['CUDA_VISIBLE_DEVICES'] = "1" os.environ['PATH'] = PATH from setuptools import setup from torch.utils.cpp_extension import BuildExtension, CUDAExtension

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

GXYM / TextBPN-Plus-Plus

仿Icdar2015数据集评估报错 #30