search

craigsapp / website-polish-scores

Front-end for digital transcriptions of the Polish Music Heritage in Open Access project
https://polishscores.org
3 stars 1 forks source link

Deal with non-UTF-7 characters in pitch range svg images #240

Closed craigsapp closed 5 months ago

craigsapp commented 5 months ago

Related to subissue https://github.com/craigsapp/website-polish-scores/issues/236#issuecomment-1895661530 where accented characters are getting messed up:

Click to view list of works that have unicode cases (beyond the basic ones that are currently being filtered out) https://polishscores.org/?id=16xx:1753 https://polishscores.org/?id=17xx:1086 https://polishscores.org/?id=17xx:1147 https://polishscores.org/?id=17xx:1150 https://polishscores.org/?id=17xx:1165 https://polishscores.org/?id=17xx:1166 https://polishscores.org/?id=17xx:1204 https://polishscores.org/?id=17xx:1205 https://polishscores.org/?id=17xx:1362 https://polishscores.org/?id=17xx:1872 https://polishscores.org/?id=17xx:1873 https://polishscores.org/?id=17xx:2349 https://polishscores.org/?id=17xx:2353 https://polishscores.org/?id=17xx:2409 https://polishscores.org/?id=17xx:353 https://polishscores.org/?id=17xx:392 https://polishscores.org/?id=17xx:436 https://polishscores.org/?id=17xx:561 https://polishscores.org/?id=17xx:972 https://polishscores.org/?id=18xx:1028 https://polishscores.org/?id=18xx:1033 https://polishscores.org/?id=18xx:1132 https://polishscores.org/?id=18xx:1174 https://polishscores.org/?id=18xx:1176 https://polishscores.org/?id=18xx:1177 https://polishscores.org/?id=18xx:1178 https://polishscores.org/?id=18xx:1180 https://polishscores.org/?id=18xx:1181 https://polishscores.org/?id=18xx:1182 https://polishscores.org/?id=18xx:1183 https://polishscores.org/?id=18xx:1184 https://polishscores.org/?id=18xx:1185 https://polishscores.org/?id=18xx:1249 https://polishscores.org/?id=18xx:1635 https://polishscores.org/?id=18xx:1636 https://polishscores.org/?id=18xx:1639 https://polishscores.org/?id=18xx:1640 https://polishscores.org/?id=18xx:1641 https://polishscores.org/?id=18xx:1643 https://polishscores.org/?id=18xx:1645 https://polishscores.org/?id=18xx:1652 https://polishscores.org/?id=18xx:1660 https://polishscores.org/?id=18xx:1661 https://polishscores.org/?id=18xx:1662 https://polishscores.org/?id=18xx:1663 https://polishscores.org/?id=18xx:1666 https://polishscores.org/?id=18xx:1668a https://polishscores.org/?id=18xx:1668b https://polishscores.org/?id=18xx:1674 https://polishscores.org/?id=18xx:1681 https://polishscores.org/?id=18xx:1690 https://polishscores.org/?id=18xx:1691 https://polishscores.org/?id=18xx:1692 https://polishscores.org/?id=18xx:1693 https://polishscores.org/?id=18xx:1694 https://polishscores.org/?id=18xx:1695 https://polishscores.org/?id=18xx:1696 https://polishscores.org/?id=18xx:1698 https://polishscores.org/?id=18xx:1699 https://polishscores.org/?id=18xx:1702 https://polishscores.org/?id=18xx:1703 https://polishscores.org/?id=18xx:1711 https://polishscores.org/?id=18xx:1712 https://polishscores.org/?id=18xx:1713 https://polishscores.org/?id=18xx:1716 https://polishscores.org/?id=18xx:1718 https://polishscores.org/?id=18xx:1720 https://polishscores.org/?id=18xx:1721 https://polishscores.org/?id=18xx:1722 https://polishscores.org/?id=18xx:1723 https://polishscores.org/?id=18xx:1758 https://polishscores.org/?id=18xx:1759 https://polishscores.org/?id=18xx:1760 https://polishscores.org/?id=18xx:1778 https://polishscores.org/?id=18xx:1800 https://polishscores.org/?id=18xx:1834 https://polishscores.org/?id=18xx:1847 https://polishscores.org/?id=18xx:194 https://polishscores.org/?id=18xx:2047 https://polishscores.org/?id=18xx:2048 https://polishscores.org/?id=18xx:2049 https://polishscores.org/?id=18xx:2050 https://polishscores.org/?id=18xx:2054 https://polishscores.org/?id=18xx:2075 https://polishscores.org/?id=18xx:2076 https://polishscores.org/?id=18xx:208 https://polishscores.org/?id=18xx:209 https://polishscores.org/?id=18xx:211 https://polishscores.org/?id=18xx:212 https://polishscores.org/?id=18xx:213 https://polishscores.org/?id=18xx:2162 https://polishscores.org/?id=18xx:2164 https://polishscores.org/?id=18xx:2176 https://polishscores.org/?id=18xx:2177 https://polishscores.org/?id=18xx:2215 https://polishscores.org/?id=18xx:2218 https://polishscores.org/?id=18xx:2222 https://polishscores.org/?id=18xx:2223 https://polishscores.org/?id=18xx:2224 https://polishscores.org/?id=18xx:2225 https://polishscores.org/?id=18xx:2228 https://polishscores.org/?id=18xx:2233 https://polishscores.org/?id=18xx:2234 https://polishscores.org/?id=18xx:2239 https://polishscores.org/?id=18xx:2250 https://polishscores.org/?id=18xx:2255 https://polishscores.org/?id=18xx:2256 https://polishscores.org/?id=18xx:2259 https://polishscores.org/?id=18xx:2262 https://polishscores.org/?id=18xx:2266 https://polishscores.org/?id=18xx:2267a https://polishscores.org/?id=18xx:2267b https://polishscores.org/?id=18xx:2273 https://polishscores.org/?id=18xx:2292 https://polishscores.org/?id=18xx:2297 https://polishscores.org/?id=18xx:2308 https://polishscores.org/?id=18xx:2309 https://polishscores.org/?id=18xx:2310 https://polishscores.org/?id=18xx:2312 https://polishscores.org/?id=18xx:2340 https://polishscores.org/?id=18xx:2363 https://polishscores.org/?id=18xx:2370 https://polishscores.org/?id=18xx:2383 https://polishscores.org/?id=18xx:2384 https://polishscores.org/?id=18xx:2389 https://polishscores.org/?id=18xx:2391 https://polishscores.org/?id=18xx:2437 https://polishscores.org/?id=18xx:2438 https://polishscores.org/?id=18xx:2439 https://polishscores.org/?id=18xx:2440 https://polishscores.org/?id=18xx:2441 https://polishscores.org/?id=18xx:2442 https://polishscores.org/?id=18xx:2443 https://polishscores.org/?id=18xx:2444 https://polishscores.org/?id=18xx:2445 https://polishscores.org/?id=18xx:2446 https://polishscores.org/?id=18xx:2458 https://polishscores.org/?id=18xx:2459 https://polishscores.org/?id=18xx:2467 https://polishscores.org/?id=18xx:276 https://polishscores.org/?id=18xx:277 https://polishscores.org/?id=18xx:285 https://polishscores.org/?id=18xx:2857 https://polishscores.org/?id=18xx:2862 https://polishscores.org/?id=18xx:2869 https://polishscores.org/?id=18xx:2954 https://polishscores.org/?id=18xx:2969 https://polishscores.org/?id=18xx:2970 https://polishscores.org/?id=18xx:2978 https://polishscores.org/?id=18xx:2979 https://polishscores.org/?id=18xx:2980 https://polishscores.org/?id=18xx:2984 https://polishscores.org/?id=18xx:2989 https://polishscores.org/?id=18xx:3023 https://polishscores.org/?id=18xx:3026 https://polishscores.org/?id=18xx:3040 https://polishscores.org/?id=18xx:3047a https://polishscores.org/?id=18xx:3047b https://polishscores.org/?id=18xx:3051 https://polishscores.org/?id=18xx:3054 https://polishscores.org/?id=18xx:3058 https://polishscores.org/?id=18xx:3059 https://polishscores.org/?id=18xx:3060 https://polishscores.org/?id=18xx:3062 https://polishscores.org/?id=18xx:3063 https://polishscores.org/?id=18xx:3068 https://polishscores.org/?id=18xx:3069 https://polishscores.org/?id=18xx:3070 https://polishscores.org/?id=18xx:3074 https://polishscores.org/?id=18xx:3095 https://polishscores.org/?id=18xx:3099 https://polishscores.org/?id=18xx:3107 https://polishscores.org/?id=18xx:3129 https://polishscores.org/?id=18xx:3212 https://polishscores.org/?id=18xx:3218 https://polishscores.org/?id=18xx:3223 https://polishscores.org/?id=18xx:3224 https://polishscores.org/?id=18xx:3225 https://polishscores.org/?id=18xx:3226 https://polishscores.org/?id=18xx:3227 https://polishscores.org/?id=18xx:3229 https://polishscores.org/?id=18xx:3230 https://polishscores.org/?id=18xx:3231 https://polishscores.org/?id=18xx:3232 https://polishscores.org/?id=18xx:3233 https://polishscores.org/?id=18xx:3234 https://polishscores.org/?id=18xx:3235 https://polishscores.org/?id=18xx:3236 https://polishscores.org/?id=18xx:3237 https://polishscores.org/?id=18xx:3239 https://polishscores.org/?id=18xx:3241 https://polishscores.org/?id=18xx:3242 https://polishscores.org/?id=18xx:3243 https://polishscores.org/?id=18xx:3244 https://polishscores.org/?id=18xx:3245 https://polishscores.org/?id=18xx:3246 https://polishscores.org/?id=18xx:3247 https://polishscores.org/?id=18xx:3248 https://polishscores.org/?id=18xx:3249 https://polishscores.org/?id=18xx:3250 https://polishscores.org/?id=18xx:3251 https://polishscores.org/?id=18xx:3252 https://polishscores.org/?id=18xx:3253 https://polishscores.org/?id=18xx:3255 https://polishscores.org/?id=18xx:3256 https://polishscores.org/?id=18xx:3257 https://polishscores.org/?id=18xx:3260 https://polishscores.org/?id=18xx:3261 https://polishscores.org/?id=18xx:3263 https://polishscores.org/?id=18xx:3264 https://polishscores.org/?id=18xx:3266 https://polishscores.org/?id=18xx:3267 https://polishscores.org/?id=18xx:3268 https://polishscores.org/?id=18xx:3269 https://polishscores.org/?id=18xx:3270 https://polishscores.org/?id=18xx:3271 https://polishscores.org/?id=18xx:3273 https://polishscores.org/?id=18xx:3274 https://polishscores.org/?id=18xx:3275 https://polishscores.org/?id=18xx:3276 https://polishscores.org/?id=18xx:3277 https://polishscores.org/?id=18xx:3278 https://polishscores.org/?id=18xx:3279 https://polishscores.org/?id=18xx:3280 https://polishscores.org/?id=18xx:3281 https://polishscores.org/?id=18xx:3282 https://polishscores.org/?id=18xx:3283 https://polishscores.org/?id=18xx:3284 https://polishscores.org/?id=18xx:3285 https://polishscores.org/?id=18xx:3286 https://polishscores.org/?id=18xx:3287 https://polishscores.org/?id=18xx:3288 https://polishscores.org/?id=18xx:3289 https://polishscores.org/?id=18xx:3290 https://polishscores.org/?id=18xx:3291 https://polishscores.org/?id=18xx:3292 https://polishscores.org/?id=18xx:3293 https://polishscores.org/?id=18xx:3294 https://polishscores.org/?id=18xx:3295 https://polishscores.org/?id=18xx:3296 https://polishscores.org/?id=18xx:3297 https://polishscores.org/?id=18xx:3298 https://polishscores.org/?id=18xx:3300 https://polishscores.org/?id=18xx:3301 https://polishscores.org/?id=18xx:3302 https://polishscores.org/?id=18xx:3303 https://polishscores.org/?id=18xx:3304 https://polishscores.org/?id=18xx:3305 https://polishscores.org/?id=18xx:3306 https://polishscores.org/?id=18xx:3307 https://polishscores.org/?id=18xx:3310 https://polishscores.org/?id=18xx:3311 https://polishscores.org/?id=18xx:3312 https://polishscores.org/?id=18xx:3313 https://polishscores.org/?id=18xx:3314 https://polishscores.org/?id=18xx:3315 https://polishscores.org/?id=18xx:3317 https://polishscores.org/?id=18xx:3318 https://polishscores.org/?id=18xx:3320 https://polishscores.org/?id=18xx:3324 https://polishscores.org/?id=18xx:3325 https://polishscores.org/?id=18xx:3327 https://polishscores.org/?id=18xx:3328 https://polishscores.org/?id=18xx:3330 https://polishscores.org/?id=18xx:3331 https://polishscores.org/?id=18xx:3333 https://polishscores.org/?id=18xx:3334 https://polishscores.org/?id=18xx:3335 https://polishscores.org/?id=18xx:3338 https://polishscores.org/?id=18xx:3340 https://polishscores.org/?id=18xx:3341 https://polishscores.org/?id=18xx:3342 https://polishscores.org/?id=18xx:3343 https://polishscores.org/?id=18xx:3344 https://polishscores.org/?id=18xx:3345 https://polishscores.org/?id=18xx:3346 https://polishscores.org/?id=18xx:3347 https://polishscores.org/?id=18xx:3348 https://polishscores.org/?id=18xx:3349 https://polishscores.org/?id=18xx:3350 https://polishscores.org/?id=18xx:3351 https://polishscores.org/?id=18xx:3352 https://polishscores.org/?id=18xx:3353 https://polishscores.org/?id=18xx:3354 https://polishscores.org/?id=18xx:3355 https://polishscores.org/?id=18xx:3356 https://polishscores.org/?id=18xx:3357 https://polishscores.org/?id=18xx:3358 https://polishscores.org/?id=18xx:3359 https://polishscores.org/?id=18xx:3360 https://polishscores.org/?id=18xx:3361 https://polishscores.org/?id=18xx:3362 https://polishscores.org/?id=18xx:3363 https://polishscores.org/?id=18xx:3365 https://polishscores.org/?id=18xx:3366 https://polishscores.org/?id=18xx:3369 https://polishscores.org/?id=18xx:3370 https://polishscores.org/?id=18xx:3374 https://polishscores.org/?id=18xx:3379 https://polishscores.org/?id=18xx:3381 https://polishscores.org/?id=18xx:3387 https://polishscores.org/?id=18xx:3430 https://polishscores.org/?id=18xx:3563 https://polishscores.org/?id=18xx:3564 https://polishscores.org/?id=18xx:3566 https://polishscores.org/?id=18xx:3567 https://polishscores.org/?id=18xx:3568 https://polishscores.org/?id=18xx:3569 https://polishscores.org/?id=18xx:3570 https://polishscores.org/?id=18xx:3571 https://polishscores.org/?id=18xx:3579 https://polishscores.org/?id=18xx:3585 https://polishscores.org/?id=18xx:3608 https://polishscores.org/?id=18xx:3609 https://polishscores.org/?id=18xx:3610 https://polishscores.org/?id=18xx:3611 https://polishscores.org/?id=18xx:3617 https://polishscores.org/?id=18xx:3619 https://polishscores.org/?id=18xx:3621 https://polishscores.org/?id=18xx:3622 https://polishscores.org/?id=18xx:3641 https://polishscores.org/?id=18xx:3642 https://polishscores.org/?id=18xx:3643 https://polishscores.org/?id=18xx:3646 https://polishscores.org/?id=18xx:3650 https://polishscores.org/?id=18xx:3653 https://polishscores.org/?id=18xx:3654 https://polishscores.org/?id=18xx:3655 https://polishscores.org/?id=18xx:3656 https://polishscores.org/?id=18xx:3657 https://polishscores.org/?id=18xx:3658 https://polishscores.org/?id=18xx:3660 https://polishscores.org/?id=18xx:3662 https://polishscores.org/?id=18xx:3667 https://polishscores.org/?id=18xx:3668 https://polishscores.org/?id=18xx:3676 https://polishscores.org/?id=18xx:3677 https://polishscores.org/?id=18xx:3678 https://polishscores.org/?id=18xx:3688 https://polishscores.org/?id=18xx:3689 https://polishscores.org/?id=18xx:3690 https://polishscores.org/?id=18xx:3699 https://polishscores.org/?id=18xx:3700 https://polishscores.org/?id=18xx:3701 https://polishscores.org/?id=18xx:3702 https://polishscores.org/?id=18xx:3703 https://polishscores.org/?id=18xx:3704 https://polishscores.org/?id=18xx:3705 https://polishscores.org/?id=18xx:3706 https://polishscores.org/?id=18xx:3762 https://polishscores.org/?id=18xx:3763 https://polishscores.org/?id=18xx:3764 https://polishscores.org/?id=18xx:3765 https://polishscores.org/?id=18xx:3766 https://polishscores.org/?id=18xx:3767 https://polishscores.org/?id=18xx:3768 https://polishscores.org/?id=18xx:3769 https://polishscores.org/?id=18xx:3770 https://polishscores.org/?id=18xx:3771 https://polishscores.org/?id=18xx:3773 https://polishscores.org/?id=18xx:3774 https://polishscores.org/?id=18xx:3775 https://polishscores.org/?id=18xx:3776 https://polishscores.org/?id=18xx:3777 https://polishscores.org/?id=18xx:3778 https://polishscores.org/?id=18xx:3780 https://polishscores.org/?id=18xx:3781 https://polishscores.org/?id=18xx:3782 https://polishscores.org/?id=18xx:3784 https://polishscores.org/?id=18xx:3785 https://polishscores.org/?id=18xx:3786 https://polishscores.org/?id=18xx:3788 https://polishscores.org/?id=18xx:3792 https://polishscores.org/?id=18xx:3974 https://polishscores.org/?id=18xx:3983 https://polishscores.org/?id=18xx:3987 https://polishscores.org/?id=18xx:3989 https://polishscores.org/?id=18xx:4007 https://polishscores.org/?id=18xx:4008 https://polishscores.org/?id=18xx:4009 https://polishscores.org/?id=18xx:4010 https://polishscores.org/?id=18xx:4012 https://polishscores.org/?id=18xx:4013 https://polishscores.org/?id=18xx:4058 https://polishscores.org/?id=18xx:4059 https://polishscores.org/?id=18xx:4060 https://polishscores.org/?id=18xx:4061 https://polishscores.org/?id=18xx:4066 https://polishscores.org/?id=18xx:4067 https://polishscores.org/?id=18xx:4068 https://polishscores.org/?id=18xx:4071 https://polishscores.org/?id=18xx:4072 https://polishscores.org/?id=18xx:4073 https://polishscores.org/?id=18xx:4075 https://polishscores.org/?id=18xx:4077 https://polishscores.org/?id=18xx:4079 https://polishscores.org/?id=18xx:4080 https://polishscores.org/?id=18xx:4081 https://polishscores.org/?id=18xx:4082 https://polishscores.org/?id=18xx:4083 https://polishscores.org/?id=18xx:4084 https://polishscores.org/?id=18xx:4086 https://polishscores.org/?id=18xx:4087 https://polishscores.org/?id=18xx:4125 https://polishscores.org/?id=18xx:4137 https://polishscores.org/?id=18xx:4150 https://polishscores.org/?id=18xx:4161 https://polishscores.org/?id=18xx:4164 https://polishscores.org/?id=18xx:4173 https://polishscores.org/?id=18xx:4174 https://polishscores.org/?id=18xx:4175 https://polishscores.org/?id=18xx:4176 https://polishscores.org/?id=18xx:4178 https://polishscores.org/?id=18xx:4179 https://polishscores.org/?id=18xx:4180 https://polishscores.org/?id=18xx:4190 https://polishscores.org/?id=18xx:4191 https://polishscores.org/?id=18xx:4199 https://polishscores.org/?id=18xx:4206 https://polishscores.org/?id=18xx:4210 https://polishscores.org/?id=18xx:4211 https://polishscores.org/?id=18xx:4212 https://polishscores.org/?id=18xx:4216 https://polishscores.org/?id=18xx:4241 https://polishscores.org/?id=18xx:4243 https://polishscores.org/?id=18xx:4248 https://polishscores.org/?id=18xx:4251 https://polishscores.org/?id=18xx:4338 https://polishscores.org/?id=18xx:683 https://polishscores.org/?id=18xx:702 https://polishscores.org/?id=18xx:891 https://polishscores.org/?id=18xx:892 https://polishscores.org/?id=18xx:893 https://polishscores.org/?id=18xx:896 https://polishscores.org/?id=18xx:927 https://polishscores.org/?id=18xx:938 https://polishscores.org/?id=19xx:115 https://polishscores.org/?id=19xx:117 https://polishscores.org/?id=19xx:118 https://polishscores.org/?id=19xx:119 https://polishscores.org/?id=19xx:120 https://polishscores.org/?id=19xx:121 https://polishscores.org/?id=19xx:122 https://polishscores.org/?id=19xx:125 https://polishscores.org/?id=19xx:126 https://polishscores.org/?id=19xx:127 https://polishscores.org/?id=19xx:17 https://polishscores.org/?id=19xx:19 https://polishscores.org/?id=19xx:2 https://polishscores.org/?id=19xx:3 https://polishscores.org/?id=19xx:38 https://polishscores.org/?id=19xx:39 https://polishscores.org/?id=19xx:4 https://polishscores.org/?id=19xx:40 https://polishscores.org/?id=19xx:43 https://polishscores.org/?id=19xx:44 https://polishscores.org/?id=19xx:45 https://polishscores.org/?id=19xx:46 https://polishscores.org/?id=19xx:5 https://polishscores.org/?id=19xx:6 https://polishscores.org/?id=19xx:64 https://polishscores.org/?id=19xx:67 https://polishscores.org/?id=19xx:9

Example case: ō which is returned as \U0000014d

https://graphemica.com/%C5%8D

This can be converted into ō or ō without leading zeros to recover the correct character.

craigsapp commented 5 months ago

Using the following regular expression on the SVG fixes the problem:

s/\/U([0-9a-fA-F]{8})/&#x$1;/gs;
craigsapp commented 5 months ago

Now fixed:

Screenshot 2024-01-21 at 18 41 37

See Baſso text which is highlighted. Crowded instrument names is a separate issue 😜

Click to view updated download script ```perl #!/usr/bin/env perl use strict; use HTTP::Request::Common; use LWP::UserAgent; my $data; while (my $line = <>) { $data .= "$line"; } my $ua = LWP::UserAgent->new; my $response = $ua->request( POST 'https://www.google.com', [ outputformat => 'svg', embedpmx => 'yes', inputdata => [$data], ] ); if ($response->is_success) { #print $response->content; my $output = $response->decoded_content; # recover Unicode characters: $output =~ s/\/U([0-9a-fA-F]{8})/&#x$1;/gs; print $output; } else { die $response->status_line; } ```