rasa-nlu-examples

[koaning]

Hi Hashir,

I just read this linkedin post. Nice work! I'm happy to hear Rasa is working well for you.

I figured reaching out to you to let you know about an effort I've started called rasa-nlu-examples(github, docs). The goal of that project is to offer more Rasa NLU tools. Some of these tools include pretrained wordembeddings/lemmatizers/tokenizers for Arabic.

Any feedback that you might on those tools would be much appreciated and I'd be all ears if there are some language tools for Arabic you'd like me to consider supporting there.

[hashirabdulbasheer]

Hi Vincent

Thats awesome. I will check it out and get back to you.

[hashirabdulbasheer]

HI Vincent

I checked out Rasa Whatlies on arabic and English using BytePairLanguage. I used pca and umap to see if similar messages cluster. But they don't. Both for English and arabic, they don't seems to cluster.

I used tweets from two categories:

CATEGORY	ARABIC	ENGLISH
TRANSFER	السلام عليكم يلزمنا حجز موعد للجوازات لاستلام اقامة بعد نقل خدمات عمالة مهنية ولكن لا اجد عند اجراء حجز الموعد مايخص استلام الاقامة	Peace be upon you. We need to book an appointment for passports to receive residency after transferring professional employment services, but I do not find when making appointment reservations about receiving residency
TRANSFER	نقل خدمات عبر ابشر ولكن من دون طباعة الاقامة، يعني لازم مراجعة الجوازات للطباعة، ايش الفايدة،اراجع الجوازات وانقل واطبع اذا كان في الاخير لازم مراجعة	Transferring services via Absher, but without printing the residency, I mean, it is necessary to review passports for printing, what is the payment, return passports, transfer and print if in the last one is necessary
TRANSFER	السلام عليكم هل بالإمكان تغيير مهنة العامل المنزلي إلى مهنه عمل أخرى لنفس الكفيل مع وجود العامل داخل المملكة. وشكرا	Peace be upon you. Is it possible to change the profession of the domestic worker to another profession for the same sponsor, with the presence of the worker inside the Kingdom? Thank you
TRAFFIC	السلام عليكم ورحمة الله وبركاتة يااخوان انا جتني مخالفة مرورية بالغلط وقدمت اعتراض وارفضوه وفي مكان بحياتي ماجيته ولاوصلته كيف الطريقة اسعفوني	Peace, mercy and blessings of God be upon you brothers. I accidentally got a traffic violation and lodged an objection and rejected it.
TRAFFIC	احاول بيع سيارتي عن طريق ابشر وعند اتمام بيع جميع البيانات تظهر هذه الجمله ( هذا الشخص لا يملك هذه السياره ) و عند مراجعه اداره مرور يفيدني الموظف ان عليها حظر نقل ملكيه من قبل الوكاله مع العلم اني انا الملك وتم مراجعه تايوتا الوكاله ولا يوجد حظر على المركبه	I am trying to sell my car through Absher, and upon completion of the sale of all the data, this sentence appears (This person does not own this car), and when reviewing the Traffic Department, the employee informs me that it must prohibit the transfer of ownership by the agency, knowing that I am the king and the Toyota agency has been reviewed and there is no ban on Vehicle
TRAFFIC	السلام عليكم تم تجديد مركبه عن طريق تطبيق الراجحي وعند نقل ملكيه المركبه في ابشر يرفض بسبب عدم السداد ؟؟	Peace be upon you. His vehicle was renewed through the Al-Rajhi application and when the ownership of the vehicle was transferred in Absher, it was rejected due to non-payment ??

For Arabic version here are the graphs

1) PCA : https://drive.google.com/file/d/1Dtdpaigzv6SqhLuL6GJT-HRK5QaZh_0z/view?usp=sharing 2) UMAP: https://drive.google.com/file/d/1WvawUrtNWNg7yzeAMCBmeC18guZGc21M/view?usp=sharing

For English:

1) PCA: https://drive.google.com/file/d/15kes9eKEnLognix_E5w9QZ_ixHjPab56/view?usp=sharing 2) UMAP: https://drive.google.com/file/d/12E5V_Q0B73nkUxdpdmIpYRb-bDcb9Hwd/view?usp=sharing

Can you check.

thanks hashir

[hashirabdulbasheer]

Here is the code that I used

tweets_ar = [
    "السلام عليكم يلزمنا حجز موعد للجوازات لاستلام اقامة بعد نقل خدمات عمالة مهنية ولكن لا اجد عند اجراء حجز الموعد مايخص استلام الاقامة",
    "نقل خدمات عبر ابشر ولكن من دون طباعة الاقامة، يعني لازم مراجعة الجوازات للطباعة، ايش الفايدة،اراجع الجوازات وانقل واطبع اذا كان في الاخير لازم مراجعة",
    "السلام عليكم هل بالإمكان تغيير مهنة العامل المنزلي إلى مهنه عمل أخرى لنفس الكفيل مع وجود العامل داخل المملكة. وشكرا",
    "السلام عليكم ورحمة الله وبركاتة يااخوان انا جتني مخالفة مرورية بالغلط وقدمت اعتراض وارفضوه وفي مكان بحياتي ماجيته ولاوصلته كيف الطريقة اسعفوني",
    "احاول بيع سيارتي عن طريق ابشر وعند اتمام بيع جميع البيانات تظهر هذه الجمله ( هذا الشخص لا يملك هذه السياره ) و عند مراجعه اداره مرور يفيدني الموظف ان عليها حظر نقل ملكيه من قبل الوكاله مع العلم اني انا الملك وتم مراجعه تايوتا الوكاله ولا يوجد حظر على المركبه",
    "السلام عليكم تم  تجديد مركبه عن طريق تطبيق الراجحي وعند نقل ملكيه المركبه في ابشر يرفض بسبب عدم السداد ؟؟"
]

tweets_en = [
    "Peace be upon you. We need to book an appointment for passports to receive residency after transferring professional employment services, but I do not find when making appointment reservations about receiving residency",
    "Transferring services via Absher, but without printing the residency, I mean, it is necessary to review passports for printing, what is the payment, return passports, transfer and print if in the last one is necessary",
    "Peace be upon you. Is it possible to change the profession of the domestic worker to another profession for the same sponsor, with the presence of the worker inside the Kingdom? Thank you",
    "Peace, mercy and blessings of God be upon you brothers. I accidentally got a traffic violation and lodged an objection and rejected it.",
    "I am trying to sell my car through Absher, and upon completion of the sale of all the data, this sentence appears (This person does not own this car), and when reviewing the Traffic Department, the employee informs me that it must prohibit the transfer of ownership by the agency, knowing that I am the king and the Toyota agency has been reviewed and there is no ban on Vehicle",
    "Peace be upon you. His vehicle was renewed through the Al-Rajhi application and when the ownership of the vehicle was transferred in Absher, it was rejected due to non-payment ??"
]

# lang_bp_ar = BytePairLanguage("ar", dim=300, vs=200_000)
lang_bp_en = BytePairLanguage("en", dim=300, vs=200_000)

embset = lang_bp_en[tweets_en]
p1 = (
    embset.transform(Pca(2))
    .plot_interactive(title="pca")
    .properties(width=500, height=500)
)
p2 = (
    embset.transform(Umap(2))
    .plot_interactive(title="umap")
    .properties(width=500, height=500)
)
p1.show()
p2.show()