Unicode emoji parser implemented in python to output JSON

[pavinjosdev]

New Unicode 15.1 emojis have been parsed to JSON data files. Parser implemented in Python. But unable to import the JSON data in emojiCategory.js file. @FelipeFTN Hope you can make the JSON import work. It's throwing an error with the fetch request I tried.

[FelipeFTN]

Heyy @pavinjosdev, great news! :rocket: I just did some coding here and added a bit to your branch! Here's my pull request, buddy! Can you review it please? https://github.com/pavinjosdev/Emoji-Copy/pull/1 I achieved to make the JSON import works properly here! Also, I removed all the gigantic JSON files and made the python parser download and process and transform the files when it's executed! Plus, now the parser.py is part of the building process! :hammer_and_wrench: :tada: I just added it to the Makefile The emojis are compiled/generated in Makefile during the building process :100:

Take a look at the Pull Request in your Fork and tell me what you think of it! Your code is just amazing, tho! :1st_place_medal: The parser idea was so great! :monster: Congrats!

Also, I almost forgot! :sweat_smile: Is there any way we can get rid of the emoji_map.py file? It's pretty big, isn't it? I don't know how to achieve this, but making it a JSON file should be so nice tho! I don't know if that's possible, or if it's the best of the ideas, since it may cause a slower execution time or something... Anyways, it's completely up to you to choose if it's best to keep it, or vanish it! :100:

Also, just appeared some issues now... Emoji Pop-up is resizing because the description texts are very long... We may fix this by changing some CSS behaviors or by making the description texts more... concise. Also, I noticed it's taking a bit too long to load some emoji tabs, can you test it on your computer too? We can keep fixing these bugs in this Pull Request, no problems on that! I'm pretty sure you may find some more for we to fix during your tests!

I just added the new debug feature into Makefile, so you can run make debug and it should be pretty easier to test the extension now! :tada:

[pavinjosdev]

@FelipeFTN Thank you very much for your help in fixing and standardizing everything 💯 I have improved parser.py as you suggested to make emoji_map load from a JSON file. To get rid of the slow loading of Peoples & Clothing tab, I skipped all the skin tone emojis.

Now to make this PR complete, the following are required:

1. Remove showing the skin tone options in Emoji-Copy UI
2. Some of the emojis have a longer description that causes Emoji-Copy UI to resize. This issue is not as big a problem as before as most of the really long descriptions were for the skin tone specific emojis. But if you could fix it in CSS or just truncate the display without affecting the search function that would be awesome 😊

FWIW, the gender and skin tone options are not working for me in the latest stable version of Emoji-Copy anyway, so if you completely stripped out that functionality then both [1] and [2] would be fixed in a single stroke 😎

[FelipeFTN]

@pavinjosdev Nice updates! :100: I don't think there is much sense in adding a new Unicode feature, but removing significant part of the emojis at the same time, don't you think? :confused: Well, maybe our approach was a little too much optimistic hahaha

So, JSON doesn't seem to work so well. I think we might look for better approaches... Thinking here, SQLite should be the way! :rocket: As I know, it's pretty fast, and should deliver the emojis blazingly fast! :zap:

I will create another branch with new changes and experiment some stuff here... Feel free to keep me in touch in case you have any idea or better solution! :bulb:

As for the skin tone, we should work on it as soon as possible, and I think this pull request is the best chance for this! :rocket: I could think of many solutions using JSON to try to solve it... Your last commits gave me a nice hint: We can filter the emojis by skin tone, and group them in a different JSON file, then apply the filter when we consult the file to get the skin tones. I didn't develop this idea so well because I think that implementing the SQLite should be a better solution at all!

_PS: Also, nice work changing from urllib to requests! :1st_placemedal:

[pavinjosdev]

@FelipeFTN SQLite is a good idea. It will make querying easier rather than having a bunch of JSON files. But it will also increase the complexity. 🛠 I also wonder whether the Linux page cache might just decide to cache the whole SQLite DB on frequent usage. Certainly it's performing some sort of caching for the JSON files as the slowness happens only on the first load. Subsequent loads are fast.

If you could find some way to preload the JSON (for example when gnome loads the extension) it would make this problem irrelevant. 🚀 Either way I don't think we have to worry about Emoji-Copy using 200kB of additional memory when gnome-shell on my rather light install with all of 3 extensions is using around 500MB. 😂

As for the skin tones, I have a rather controversial take as I see it as pointless. The cheesy yellow tone 🧀 is perfect for emojis to express emotions, why create skin tones which are really just approximations to further divide people into clearly artificial categories. What next? Will they come up with tones for the eyes & hair. Next it will be full RGB for everything after we find purple aliens or someone is offended the existing tones don't represent them 😩

[FelipeFTN]

Hey, @pavinjosdev! I did some tests here, and I had such a success while testing the SQLite structure for parser.py! :tada: It's still under development, I will open a Pull Request as soon as it gets done! Here are the changes in case you want to take a look: https://github.com/pavinjosdev/Emoji-Copy/compare/master...FelipeFTN:Emoji-Copy:sqlite

I'm still implementing the Javascript part, so it's still a work in progress :+1: I will take some rest now, I will keep the work later! :sleeping: I will try to open the pull request today's night! :night_with_stars:

I managed to merge the emoji_map.json file into the unicode.org's file, so we have the best of both worlds! :100: Take a look at the parser.js later, it's getting amazing! Also, the file was not that big, with more or less 500 Kb Not that bad, even considering we will just make Queries and not gonna load everything directly on memory! Also I could classify the emojis per Group and Skin tone :+1: So, it will be a lot easier to fix the feature later on.

About the skin tones, yeah, I don't know many people who actually use it, but I think the Project should deliver all the emojis available as possible (no chance to concurrency hahaha)! It will be a nice challenge to have this much of emojis without crashing or lagging someone's PC, don't you think? It will be pretty fun to see it working flawlessly and blazingly fast tho! haha! :smile:

[FelipeFTN]

@pavinjosdev The SQLite implementation is working flawlessly!! :tada: https://github.com/pavinjosdev/Emoji-Copy/pull/2 I achieved to keep the skin tones during the searches, but removed them to show up over categories tabs, to avoid memory overload :space_invader: Take a look!

[FelipeFTN]

@pavinjosdev For the extension to work properly, the user will need to install libgda, which is used to connect SQLite.

You need libgda for this extension to work.

Fedora: sudo dnf install libgda libgda-sqlite

Arch Linux: sudo pacman -S libgda6

Ubuntu/Debian: sudo apt install gir1.2-gda-5.0

openSUSE: sudo zypper install libgda-6_0-sqlite

[pavinjosdev]

@FelipeFTN Thanks, I did install the dependencies and it's working great on Fedora and Arch. But gnome-shell on OpenSuse is segfaulting with libgda enabled. Here's my bug report to OpenSuse with all the details: https://bugzilla.opensuse.org/show_bug.cgi?id=1219970

[pavinjosdev]

@FelipeFTN I'm not holding up much hope that the OpenSuse Gnome team would get to this bug any time soon 🥺 Would it be possible to remove the GDA dependency and use an SQLite library we ship with Emoji-Copy, something along the lines of sql.js ?

Also, the search could use some improvement. I'm thinking of using something like FuseJs for fuzzy search. For example, currently I type in "roll eye" and the "face with rolling eyes" emoji (🙄) does not show up.

[FelipeFTN]

Heyy @pavinjosdev! Nice work with the bug report! :space_invader: How are you testing stuff on OpenSuse? Are Wayland & Gnome-Shell up to date?

Here's a few steps to test the extension with a better debug logger that I use to debug Emoji Copy here: 1. Start the system with X11 (you can change it during log in) 2. Make sure this branch's extension version is installed, and not the current one from the gnome extensions store 3. Run the debug session by running: dbus-run-session -- gnome-shell --nested --wayland or make debug from the Emoji-Copy root directory, which will execute the Makefile, clean, build, install and debug. 4. A new window may open with a wayland session running, and you will see at the terminal, all the logs related to that session + extension error logs.

The advantage of running this way in X11 instead of using Wayland is that you don't need to restart the session every time, just run the command showed above. Also, the current session usually won't crash during your tests. I'm afraid this is not a OpenSuse's system problem, but maybe some error in our code

Please, keep me updating me with your findings! :mag_right: Also, feel free to try implementing the https://github.com/sql-js/sql.js/ as you said, or any better solution you may find! I think sticking with SQLite is always a better idea for the future, but I'm open to new solutions for this problem. :100:

[pavinjosdev]

@FelipeFTN Thank you very much for the debug instructions. It makes development much easier. 🚀

Wayland and gnome-shell are both up to date. ☘

Of course gnome-shell crashed within the nested wayland session too, at least it didn't make the main X11 session unresponsive:

pavin@localhost:~/Emoji-Copy> make debug
[-] COMPILING SCHEMA...
[+] SCHEMA COMPILED
[+] Fetching unicode data from https://unicode.org/Public/emoji/15.1/emoji-test.txt
[+] Loading existing emojis from emoji_map.json to database... 🍳
[+] Finished loading existing emojis to database! 🎉
[+] Loading official unicode emojis... 🍳
[+] Finished loading official unicode emojis to database! 🎉
[!] Emoji Count: 5037
[!] run `du -ah ./emoji-copy@felipeftn/data/` to get the DBs current size.
[-] ZIPPING EMOJI COPY...
  adding: LICENSE (deflated 65%)
  adding: data/ (stored 0%)
  adding: data/.gitkeep (stored 0%)
  adding: data/emojis.db (deflated 77%)
  adding: emojiButton.js (deflated 69%)
  adding: emojiCategory.js (deflated 64%)
  adding: emojiOptionsBar.js (deflated 67%)
  adding: emojiSearchItem.js (deflated 67%)
  adding: emojis.db (deflated 77%)
  adding: extension.js (deflated 71%)
  adding: files.js (deflated 43%)
  adding: handlers/ (stored 0%)
  adding: handlers/files.js (deflated 43%)
  adding: handlers/sqlite.js (deflated 60%)
  adding: icons/ (stored 0%)
  adding: icons/emoji-activities-symbolic.svg (deflated 50%)
  adding: icons/emoji-body-symbolic.svg (deflated 69%)
  adding: icons/emoji-flags-symbolic.svg (deflated 67%)
  adding: icons/emoji-food-symbolic.svg (deflated 51%)
  adding: icons/emoji-nature-symbolic.svg (deflated 67%)
  adding: icons/emoji-objects-symbolic.svg (deflated 64%)
  adding: icons/emoji-people-symbolic.svg (deflated 41%)
  adding: icons/emoji-symbols-symbolic.svg (deflated 68%)
  adding: icons/emoji-travel-symbolic.svg (deflated 54%)
  adding: locale/ (stored 0%)
  adding: locale/ar/ (stored 0%)
  adding: locale/ar/LC_MESSAGES/ (stored 0%)
  adding: locale/ar/LC_MESSAGES/emoji-copy.mo (deflated 46%)
  adding: locale/ar/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/cs/ (stored 0%)
  adding: locale/cs/LC_MESSAGES/ (stored 0%)
  adding: locale/cs/LC_MESSAGES/emoji-copy.mo (deflated 43%)
  adding: locale/cs/LC_MESSAGES/emoji-copy.po (deflated 64%)
  adding: locale/de/ (stored 0%)
  adding: locale/de/LC_MESSAGES/ (stored 0%)
  adding: locale/de/LC_MESSAGES/emoji-copy.mo (deflated 44%)
  adding: locale/de/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/emoji-copy.pot (deflated 72%)
  adding: locale/eo/ (stored 0%)
  adding: locale/eo/LC_MESSAGES/ (stored 0%)
  adding: locale/eo/LC_MESSAGES/emoji-copy.mo (deflated 33%)
  adding: locale/eo/LC_MESSAGES/emoji-copy.po (deflated 70%)
  adding: locale/es_ES/ (stored 0%)
  adding: locale/es_ES/LC_MESSAGES/ (stored 0%)
  adding: locale/es_ES/LC_MESSAGES/emoji-copy.mo (deflated 44%)
  adding: locale/es_ES/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/fr/ (stored 0%)
  adding: locale/fr/LC_MESSAGES/ (stored 0%)
  adding: locale/fr/LC_MESSAGES/emoji-copy.mo (deflated 44%)
  adding: locale/fr/LC_MESSAGES/emoji-copy.po (deflated 67%)
  adding: locale/gl/ (stored 0%)
  adding: locale/gl/LC_MESSAGES/ (stored 0%)
  adding: locale/gl/LC_MESSAGES/emoji-copy.mo (deflated 46%)
  adding: locale/gl/LC_MESSAGES/emoji-copy.po (deflated 67%)
  adding: locale/it/ (stored 0%)
  adding: locale/it/LC_MESSAGES/ (stored 0%)
  adding: locale/it/LC_MESSAGES/emoji-copy.mo (deflated 46%)
  adding: locale/it/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/nl/ (stored 0%)
  adding: locale/nl/LC_MESSAGES/ (stored 0%)
  adding: locale/nl/LC_MESSAGES/emoji-copy.mo (deflated 44%)
  adding: locale/nl/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/oc/ (stored 0%)
  adding: locale/oc/LC_MESSAGES/ (stored 0%)
  adding: locale/oc/LC_MESSAGES/emoji-copy.mo (deflated 45%)
  adding: locale/oc/LC_MESSAGES/emoji-copy.po (deflated 66%)
  adding: locale/pl/ (stored 0%)
  adding: locale/pl/LC_MESSAGES/ (stored 0%)
  adding: locale/pl/LC_MESSAGES/emoji-copy.mo (deflated 40%)
  adding: locale/pl/LC_MESSAGES/emoji-copy.po (deflated 64%)
  adding: locale/pt_BR/ (stored 0%)
  adding: locale/pt_BR/LC_MESSAGES/ (stored 0%)
  adding: locale/pt_BR/LC_MESSAGES/emoji-copy.mo (deflated 35%)
  adding: locale/pt_BR/LC_MESSAGES/emoji-copy.po (deflated 69%)
  adding: locale/tr/ (stored 0%)
  adding: locale/tr/LC_MESSAGES/ (stored 0%)
  adding: locale/tr/LC_MESSAGES/emoji-copy.mo (deflated 44%)
  adding: locale/tr/LC_MESSAGES/emoji-copy.po (deflated 64%)
  adding: locale/zh_Hans/ (stored 0%)
  adding: locale/zh_Hans/LC_MESSAGES/ (stored 0%)
  adding: locale/zh_Hans/LC_MESSAGES/emoji-copy.mo (deflated 36%)
  adding: locale/zh_Hans/LC_MESSAGES/emoji-copy.po (deflated 62%)
  adding: metadata.json (deflated 42%)
  adding: prefs.js (deflated 67%)
  adding: schemas/ (stored 0%)
  adding: schemas/gschemas.compiled (deflated 38%)
  adding: schemas/org.gnome.shell.extensions.emoji-copy.gschema.xml (deflated 67%)
  adding: sqlite.js (deflated 60%)
  adding: stylesheet.css (deflated 67%)
deleting: locale/emoji-copy.pot
deleting: locale/ar/LC_MESSAGES/emoji-copy.po
deleting: locale/cs/LC_MESSAGES/emoji-copy.po
deleting: locale/de/LC_MESSAGES/emoji-copy.po
deleting: locale/eo/LC_MESSAGES/emoji-copy.po
deleting: locale/es_ES/LC_MESSAGES/emoji-copy.po
deleting: locale/fr/LC_MESSAGES/emoji-copy.po
deleting: locale/gl/LC_MESSAGES/emoji-copy.po
deleting: locale/it/LC_MESSAGES/emoji-copy.po
deleting: locale/nl/LC_MESSAGES/emoji-copy.po
deleting: locale/oc/LC_MESSAGES/emoji-copy.po
deleting: locale/pl/LC_MESSAGES/emoji-copy.po
deleting: locale/pt_BR/LC_MESSAGES/emoji-copy.po
deleting: locale/tr/LC_MESSAGES/emoji-copy.po
deleting: locale/zh_Hans/LC_MESSAGES/emoji-copy.po
[+] EMOJI COPY BUILT
gnome-extensions install emoji-copy@felipeftn.zip --force
Extension installed successfully! Now restart the Shell ('Alt'+'F2', then 'restart').
dbus-run-session -- gnome-shell --nested --wayland
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.Daemon' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.Daemon'
fusermount3: failed to access mountpoint /run/user/1000/gvfs: Permission denied
libmutter-Message: 22:54:10.007: Running GNOME Shell (using mutter 45.3) as a Wayland display server
libEGL warning: DRI2: failed to authenticate
MESA: error: ZINK: vkCreateInstance failed (VK_ERROR_INCOMPATIBLE_DRIVER)
libEGL warning: egl: failed to create dri2 screen
libmutter-Message: 22:54:10.145: Using public X11 display :1, (using :2 for managed services)
libmutter-Message: 22:54:10.145: Using Wayland display name 'wayland-0'
Window manager warning: Failed to set environment variable GNOME_SETUP_DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
Window manager warning: Failed to set environment variable DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
Window manager warning: Failed to set environment variable XAUTHORITY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
Window manager warning: Failed to set environment variable WAYLAND_DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
GNOME Shell-Message: 22:54:10.459: Unset XDG_SESSION_ID, getCurrentSessionProxy() called outside a user session. Asking logind directly.
GNOME Shell-Message: 22:54:10.459: Will monitor session 1
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.freedesktop.systemd1' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.Shell.Screencast' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Activated service 'org.freedesktop.systemd1' failed: Process org.freedesktop.systemd1 exited with status 1
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.freedesktop.impl.portal.PermissionStore' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.Shell.CalendarServer' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.freedesktop.impl.portal.PermissionStore'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.evolution.dataserver.Sources5' requested by ':1.4' (uid=1000 pid=12428 comm="/usr/libexec/gnome-shell/gnome-shell-calendar-serv")
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.Shell.Notifications' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")

** (gnome-shell:12378): WARNING **: 22:54:10.639: Could not issue 'GetUnit' systemd call
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.OnlineAccounts' requested by ':1.5' (uid=1000 pid=12439 comm="/usr/libexec/evolution-data-server/evolution-sourc")
GNOME Shell-Message: 22:54:10.686: Failed to launch ibus-daemon: Failed to execute child process “ibus-daemon” (No such file or directory)
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.evolution.dataserver.Sources5'
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.Shell.CalendarServer'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.evolution.dataserver.Calendar8' requested by ':1.4' (uid=1000 pid=12428 comm="/usr/libexec/gnome-shell/gnome-shell-calendar-serv")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.Shell.Notifications'
goa-daemon-Message: 22:54:10.754: goa-daemon version 3.48.0 starting
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.Identity' requested by ':1.8' (uid=1000 pid=12468 comm="/usr/libexec/goa-daemon")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.OnlineAccounts'
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.Identity'

(process:12488): libgoaidentity-WARNING **: 22:54:10.798: GoaKerberosIdentityManager: Using polling for change notification for credential cache type 'KEYRING'
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.evolution.dataserver.Calendar8'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gnome.evolution.dataserver.AddressBook10' requested by ':1.7' (uid=1000 pid=12476 comm="/usr/libexec/evolution-data-server/evolution-calen")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gnome.evolution.dataserver.AddressBook10'
GNOME Shell-Message: 22:54:10.896: Error looking up permission: GDBus.Error:org.freedesktop.portal.Error.NotFound: No entry for geolocation
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.UDisks2VolumeMonitor' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.UDisks2VolumeMonitor'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.GoaVolumeMonitor' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.GoaVolumeMonitor'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.AfcVolumeMonitor' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.AfcVolumeMonitor'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.MTPVolumeMonitor' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.MTPVolumeMonitor'
dbus-daemon[12377]: [session uid=1000 pid=12377] Activating service name='org.gtk.vfs.GPhoto2VolumeMonitor' requested by ':1.0' (uid=1000 pid=12378 comm="gnome-shell --nested --wayland")
dbus-daemon[12377]: [session uid=1000 pid=12377] Successfully activated service 'org.gtk.vfs.GPhoto2VolumeMonitor'
make: *** [Makefile:35: debug] Error 139

(evolution-calendar-factory:12476): libedbus-private-WARNING **: 22:54:12.100: Error setting property 'ConnectionStatus' on interface org.gnome.evolution.dataserver.Source: The connection is closed (g-io-error-quark, 18)

(evolution-addressbook-factory:12501): libedbus-private-WARNING **: 22:54:12.101: Error setting property 'ConnectionStatus' on interface org.gnome.evolution.dataserver.Source: The connection is closed (g-io-error-quark, 18)
gnome-shell-calendar-server[12428]: Lost (or failed to acquire) the name org.gnome.Shell.CalendarServer - exiting
A connection to the bus can't be made
libEGL warning: DRI2: failed to authenticate
pavin@localhost:~/Emoji-Copy> MESA: error: ZINK: vkCreateInstance failed (VK_ERROR_INCOMPATIBLE_DRIVER)
libEGL warning: egl: failed to create dri2 screen
MESA: error: ZINK: vkCreateInstance failed (VK_ERROR_INCOMPATIBLE_DRIVER)
glx: failed to create drisw screen
failed to load driver: zink

(org.gnome.Shell.Screencast:12414): GLib-GIO-CRITICAL **: 22:54:12.207: Error while sending AddMatch() message: The connection is closed

(org.gnome.Shell.Screencast:12414): GLib-GIO-CRITICAL **: 22:54:12.208: Error while sending AddMatch() message: The connection is closed

(org.gnome.Shell.Screencast:12414): GLib-GIO-CRITICAL **: 22:54:12.208: Error while sending AddMatch() message: The connection is closed

(org.gnome.Shell.Screencast:12414): GLib-GIO-CRITICAL **: 22:54:12.208: Error while sending AddMatch() message: The connection is closed

(org.gnome.Shell.Screencast:12414): Gjs-CRITICAL **: 22:54:12.209: JS ERROR: Gio.IOErrorEnum: Error calling StartServiceByName for org.gnome.Mutter.ScreenCast: The connection is closed
_injectToMethod/klass[method]@resource:///org/gnome/gjs/modules/core/overrides/Gio.js:292:25
wrapper@resource:///org/gnome/gjs/modules/core/overrides/Gio.js:253:17
ScreencastService@resource:///org/gnome/Shell/Screencast/js/screencastService.js:437:23
main@resource:///org/gnome/Shell/Screencast/js/main.js:11:9
@file:///usr/share/gnome-shell/org.gnome.Shell.Screencast:9:7

(org.gnome.Shell.Screencast:12414): Gjs-CRITICAL **: 22:54:12.209: Module file:///usr/share/gnome-shell/org.gnome.Shell.Screencast threw an exception
^C

I continue my efforts in debugging... 🐢

[pavinjosdev]

@FelipeFTN I've ported Emoji-Copy to use sqljs instead of GDA. Tested on Wayland and X11 sessions, no issues. Can you test and approve the PR?

Bugs found:

• The options bar to choose gender and skin tone does nothing. It's also broken in the current stable version installed from gnome extensions, so not a regression or bug introduced with this PR.

[pavinjosdev]

Btw, you can remove the GDA dependencies from emoji-copy@felipeftn/metadata.json 😅

[FelipeFTN]

Heyy @pavinjosdev!! Really nice working on this Gda migration to SQL.js library! :partying_face: It's working amazingly great here!

I just updated some stuff on the code, take a look: https://github.com/pavinjosdev/Emoji-Copy/pull/3! :tada:

Also, can you check for the sqlite.py replace query? I don't know if it's working properly on emoji updating. There are many duplicated emojis, like transgender flag, or rainbow flag... Can you take a look at this issue please?

Despite that, really nice work! :rocket:

[pavinjosdev]

I just updated some stuff on the code, take a look: pavinjosdev#3! 🎉

@FelipeFTN Great work tidying up everything! 😌✨

Also, can you check for the sqlite.py replace query? I don't know if it's working properly on emoji updating. There are many duplicated emojis, like transgender flag, or rainbow flag... Can you take a look at this issue please?

This was not an SQL issue, but due to parsing similar looking emojis with different codepoints from the unicode test file. Search for E4.0 rainbow flag and you will find two, one of which is unqualified. 😅 This and a bug with skin tone parsing has been fixed. I loaded up the DB into an online SQLite viewer and gave it good look 👀 just to make sure there are no more surprises!

[FelipeFTN]

@pavinjosdev looks like we're good to go, don't you think? :rocket: Emoji tone selector buttons is not working yet, but that's no problem for now... I shall create a new Pull Request solving this issue soon! :+1:

Searching skin tones seems to be working properly and we finished our work! @pavinjosdev Do you have any other change or polishment to do, or this is it?! :star_struck:

[pavinjosdev]

@FelipeFTN Now that I think about it, I will really regret not having fuzzy search in this PR. Search works but is not very user friendly right now, for example I search eye and only 👁 shows up, to get 👀 I have to spell out eyes 😬

I think I will add this and you can have a final look over 😘

[pavinjosdev]

@FelipeFTN It's all done, finally. ✨ Search has been improved, but not with fuzzy search as sql.js was not compiled with the spellfix1 sqlite extension. No problem, it works as a normal user would expect. Searching "eye" shows both emojis: 👁 and 👀 Search for "eye roll" and "roll eye" both gives the desired emoji: 🙄

Let's go! 🚀

[FelipeFTN]

Heyyyy @pavinjosdev!!! Nice changes buddy! I loved it! :rocket: :tada: The search is 100x better! I really think this is ready to go! I will be preparing for publishing, then I'll merge the changes to main!

Thank you so much for your help with this! You did a really nice work!! :handshake:

[pavinjosdev]

Heyyyy @pavinjosdev!!! Nice changes buddy! I loved it! 🚀 🎉 The search is 100x better! I really think this is ready to go! I will be preparing for publishing, then I'll merge the changes to main!

Thank you so much for your help with this! You did a really nice work!! 🤝

Thank you very much for the kind words ☺️🥰

[FelipeFTN]

@pavinjosdev can you merge this please? :pray: https://github.com/pavinjosdev/Emoji-Copy/pull/4 I'm having some trouble to get our changes approved by the Gnome Extensions Maintainers :confused:

Here's the last email I received from them:

FelipeFTN's extension, "Emoji Copy", version 19 has a new review:

JustPerfection posted a review on February 27, 2024:

1. `async enable()` is fine since the extension system call is `async`.

2. Missed line 5    `emojiButton.js` before. Please move that to the functions instead of calling on global scope.

3. Can you open an issue on [gjs repo](https://gitlab.gnome.org/GNOME/gjs)?
    Because IMO allowing that large 3rd party library for a bug in `gda` is too much.

You can see the reviews here:

• https://extensions.gnome.org/review/51340
• https://extensions.gnome.org/review/51346

[pavinjosdev]

@FelipeFTN Merged, we can replace sql.js with the non-minified version from their releases. There have been some comments on the opensuse bug report, it seems upstream (gnome) is already aware of this bug: https://gitlab.gnome.org/GNOME/libgda/-/issues/267

I see you've already requested an exception to use the minified version, hope they grant it because there have been no updates to the 2 month old libgda issue from Gnome's end.

[FelipeFTN]

Hey @pavinjosdev! I argued with the Gnome Extensions Maintainer about using the non-minified version. It might not be the best idea, we are not changing, editing or reading the sql.js library anyway, doesn't seem to be a problem to keep using a minified version of it. Also, non-minified version is not recommended for "production use", since it's only recommended for debugging purposes. The file is pretty big - almost 4 megabytes of pure JavaScript - counting more than 130k lines of code. :confused:

Here is my answer to Gnome Extensions:

[...]
Another thing, I could perform all of your requests but one.
I understand your comprehension of the Gnome Extensions is just perfect, but I have some questions here about this suggestion you sent:
- 4. If you want to send a third party library, don't use the minified version (`libs/sql/sql.js`);
    [EGO Review Guidelines: Obfuscated Code](https://gjs.guide/extensions/review-guidelines/review-guidelines.html#code-must-not-be-obfuscated)
"Extension code MUST be readable and reviewable JavaScript".

I know it is important to keep the extension code clean and understandable, no doubt on that.
But the third party library is kinda a different thing. The non-minified code is recommended only for debug purposes by the code owners themselves.
Using the non-minified version would directly impact the performance of the extension, since the non-minified code has more than 130k lines of Javascript code.
 I'm here to put my considerations over this topic of you to analyze it, since you have the mastery and complete understanding of the development of extensions, I know you will make the best decision, and we will follow through with it.
Since it's a third party library, as I see, it is not a core part of the extension's code. Meaning that it was not written by the extension owner and its code should not be read, changed or debugged with time.
It is a lot better for me, and the extension to run in the minified (production) version of the SQL.js library.
My plea is that we can continue publishing the extension in this way, and in the future perhaps, find a better way to work with this library or some other.

Thank you for your attention & help.
- FelipeFTN

Thank you for advising me about the already created issue on libgda for OpenSuse on Gnome Repository, I will send this information to the reviser again and see what we got. :crossed_fingers: Thank you for your help again, @pavinjosdev! I will keep you updated!

[pavinjosdev]

@FelipeFTN Okay, thank you 😻

Some things to consider:

• sql.js is a very popular library, I think the only changes I've made are on:
  1. Line 9: added export to var initSqlJs function
  2. Line 27: added var when initializing variable module = undefined;

Gnome's JS interpreter was throwing an error without those. The maintainer can do a diff to verify. 🤓

[pavinjosdev]

@FelipeFTN How are things coming along? I registered for a Gnome extensions account but it seems I'm not able to comment on the discussion: https://extensions.gnome.org/review/51346

Somewhere along the way my extension manager decided to replace the perfectly working copy of Emoji-Copy v19 with the stable version from gnome extensions, so I've lost all the new emojis. 🙈

Waiting for some resolution to upstream's complaints 😉

[FelipeFTN]

I will email them today, @pavinjosdev! Hopefully they allow us to deploy like this, with the minified sql.js version :crossed_fingers:

You can try to update or change the extension name in metadata.json, doing, so the extension manager will not find the extension online, and therefore, will not update. Another approach is to update the version in metadata.json to a new one - one that doesn't exist yet - like 20 or something. :broom:

I will keep you updated from here! :100:

[pavinjosdev]

@FelipeFTN Thanks, updating the version seems to have worked so far. But I think I found a bug causing the process to be OOM killed. The extension manager has disabled Emoji-Copy with the following reason after a few days:

Mar 12 08:42:08 suse-pc gnome-shell[3173]: Aborted(OOM)
Mar 12 08:42:08 suse-pc gnome-shell[3173]: Extension emoji-copy@felipeftn: Error: Aborted(OOM). Build with -sASSERTIONS for more info.
                                             C@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:68:142
                                             p@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:125:458
                                             Xh@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:112912
                                             ye@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:167089
                                             FG@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:7785
                                             Yb@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:472222
                                             vG@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:251598
                                             tC@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:371032
                                             ld@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:16919
                                             tf@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:112098
                                             Vo@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:245395
                                             Wo@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:5963
                                             ve@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:107578
                                             hf@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:395773
                                             _e@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:21472
                                             XI@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:175885
                                             $b@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:474003
                                             ae@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:1951
                                             Gc@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:358
                                             ej@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:55:39148
                                             Pv@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:260705
                                             vk@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:29190
                                             lh@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:62:19132
                                             initSqlJs/initSqlJsPromise</f._sqlite3_prepare_v2@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:130:414
                                             initSqlJs/initSqlJsPromise</f.onRuntimeInitialized/e.prototype.exec@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:43:513
                                             query@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/handlers/sql.js:73:25
                                             select_by_group@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/handlers/sql.js:61:17
                                             EmojiCategory@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/emojiCategory.js:37:41
                                             _createAllCategories@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/extension.js:244:33
                                             enable@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/extension.js:74:10

[pavinjosdev]

Seems like the sql.js library is the problem.. rewrite it with JSON? 🥲

[FelipeFTN]

No way! :eyes: hahaha
@pavinjosdev, Is this right? No external problems involved? Can you take a look at the RAM memory monitor? I did a quick search here, and this might be related to memory allocation issues.

Can you, please, test again with a different version of sqlite.js files from the library? Perhaps older versions, or different ones? Please, update me on this! :envelope:

I would love to see this feature released really soon! :sparkles:

[pavinjosdev]

@FelipeFTN I have sysstat (SAR) running on my machine. There were no general memory exhaustion around this time but there was an incident with Thunderbird going crazy and hogging up all the disk I/O resources that did end up causing the whole system to slow way down and lag. This was fixed and the problem (with Emoji-Copy or Tbird) has not recurred. I will continue to monitor for any future issues. 🤞😉

[pavinjosdev]

@FelipeFTN It has unfortunately happened again without any other memory/CPU/disk I/O exhaustion.

Mar 16 08:58:45 suse-pc gnome-shell[3222]: Extension emoji-copy@felipeftn: Error: Aborted(OOM). Build with -sASSERTIONS for more info.

                                           Stack trace:
                                             C@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:68:142
                                             p@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:125:458
                                             Xh@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:112912
                                             ye@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:167089
                                             FG@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:7785
                                             Yb@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:472222
                                             vG@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:251598
                                             tC@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:371032
                                             ld@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:16919
                                             tf@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:112098
                                             Vo@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:245395
                                             Wo@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:5963
                                             ve@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:107578
                                             hf@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:395773
                                             _e@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:61:21472
                                             XI@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:175885
                                             $b@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:474003
                                             ae@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:1951
                                             Gc@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:56:358
                                             ej@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:55:39148
                                             Pv@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:57:260705
                                             vk@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:60:29190
                                             lh@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/libs/sql/sql.js:62:19132
                                             initSqlJs/initSqlJsPromise</f._sqlite3_prepare_v2@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/li>
                                             initSqlJs/initSqlJsPromise</f.onRuntimeInitialized/e.prototype.exec@file:///home/pavin/.local/share/gnome-shell/extensions/emoji>
                                             query@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/handlers/sql.js:73:25
                                             select_by_group@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/handlers/sql.js:61:17
                                             EmojiCategory@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/emojiCategory.js:37:41
                                             _createAllCategories@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/extension.js:244:33
                                             enable@file:///home/pavin/.local/share/gnome-shell/extensions/emoji-copy@felipeftn/extension.js:74:10
                                             async*_callExtensionEnable@resource:///org/gnome/shell/ui/extensionSystem.js:253:38
                                             _enableAllExtensions@resource:///org/gnome/shell/ui/extensionSystem.js:771:24
                                             async*_sessionUpdated@resource:///org/gnome/shell/ui/extensionSystem.js:800:20
                                             async*ExtensionManager/<@resource:///org/gnome/shell/ui/extensionSystem.js:44:18
                                             _callHandlers@resource:///org/gnome/gjs/modules/core/_signals.js:130:42
                                             _emit@resource:///org/gnome/gjs/modules/core/_signals.js:119:10
                                             _sync@resource:///org/gnome/shell/ui/sessionMode.js:212:14
                                             popMode@resource:///org/gnome/shell/ui/sessionMode.js:179:14
                                             _continueDeactivate@resource:///org/gnome/shell/ui/screenShield.js:544:30

I will test with other versions of sql.js as previously suggested.

[pavinjosdev]

Fixed closing the DB in emoji-copy@felipeftn/handlers/sql.js -> destroy() 🤞

[FelipeFTN]

Wooow!! Nice work @pavinjosdev! :tada: I just forgot closing the db connection before cleaning the state hahaha :man_facepalming: :sweat_smile:

Nice work! I hope this will make this feature ready to launch... I will send another email to Gnome Extensions today! Hope we can launch it using sql lib!! :pray:

[FelipeFTN]

We did it, @pavinjosdev!! :partying_face: :tada: Finally, the Emoji Copy version 20 is now launched!! :rocket:

Please, approve this final polishments before we merge it into main, and officially do the v20 release! :100:

• https://github.com/pavinjosdev/Emoji-Copy/pull/6

[pavinjosdev]

Thank you very much @FelipeFTN 💙, I know the hoops you had to jump through getting the extension approved with upstream 🥹. So I'm going to do a most inappropriate jig now.

[compacted-mug-dropper]

Daaaammmnnn, just got the email this was merged, amazing work fellas