[Proposal] Migrate keras to tf.keras

BrikerMan commented 5 years ago

I am proposing to change keras to tf.keras for better performance, better serving and add TPU support. Maybe we should re-write the whole project, clean up the code, add missing documents and so on.

Here are the features I am planning to add.

Multi-GPU/TPU support
Export model for Tensorflow Serving
Fine-tune Ability for W2V and BERT

alexwwang commented 5 years ago

Models: BiLSTM_Model CNN_LSTM_Model AVCNN_Model KMaxCNN_Model RNN_CNN_Model AVRNN_Model DropoutBiGRU_Model DPCNN_Model

Layers: AttentionWeightedAverage KMaxPooling CRF

BrikerMan commented 5 years ago

Models: BiLSTM_Model CNN_LSTM_Model AVCNN_Model KMaxCNN_Model RNN_CNN_Model AVRNN_Model DropoutBiGRU_Model DPCNN_Model

Layers: AttentionWeightedAverage KMaxPooling CRF

Look good to me. Let's use this.

BrikerMan commented 5 years ago

@alexwwang Are u working on classification models, If not I am planning to work on classifications.

alexwwang commented 5 years ago

Yes. And you may write test for the new models. Thanks in advance.

On Mon, 24 Jun 2019 at 16:42, Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang Are u working on classification models, If not I am planning to work on classifications.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKX3IDENH6JDMAEQTZ3P4CCHLA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYMGXTY#issuecomment-504916943, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKUPWNSCHIZ35C647TLP4CCHLANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

@alexwwang @HaoyuHu How about releasing a new version with package name kashgari-tf, and update keras version with name kashgari. I tried that and it works, check it out. https://pypi.org/project/kashgari-tf/

haoyuhu commented 5 years ago

@BrikerMan I have fix a bug in https://github.com/BrikerMan/Kashgari/pull/108, but that part of the code was modified in 9e72bbb6060449cd4562b030b2996a4a70eed4f6. Then the problem reappears.

reproduce:

import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)

BrikerMan commented 5 years ago

@HaoyuHu Sorry man, it must be a mistake.

alexwwang commented 5 years ago

So that means to keep developing original Keras edition with future features?

Any routine plan or consideration to share?

Let me give out a brick to bring golds. I think the better idea is to make an abstract up-level program to define embeddings, layers, models even optimizers and data processors, and build a midware to stick keras backend, and tf-keras to the up-level program.

Thus we don't have to maintain two projects, to do duplicate works in each branch.

How do you think?

On Wed, Jun 26, 2019, 20:36 Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu https://github.com/HaoyuHu How about releasing a new version with package name kashgari-tf, and update keras version with name kashgari. I tried that and it works, check it out. https://pypi.org/project/kashgari-tf/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKTBIC6XJOFKOVJPF2DP4NPFFA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYTMAEI#issuecomment-505856017, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKV3LF5SZIMTDEIN4CTP4NPFFANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

@alexwwang No, I think we should keep the old version so that we will not break the user's code. keras version will be maintained for a while without new features.

Supporting both keras and tf.keras sounds good. But I am worried it will cost lots of time to maintain.

alexwwang commented 5 years ago

@BrikerMan I agree with you now. The proposal I drew this afternoon appears to be too ideal and I'm afraid it won't ever come true in that simple way.

BrikerMan commented 5 years ago

@BrikerMan I agree with you now. The proposal I drew this afternoon appears to be too ideal and I'm afraid it won't ever come true in that simple way.

Yes, now the only thing we should discuss is where to use the package name kashgari-tf.

alexwwang commented 5 years ago

I am not sure yet. I need some time to figure out the reasons to those known and unknown errors first.

On Thu, 27 Jun 2019 at 19:00, Eliyar Eziz notifications@github.com wrote:

@BrikerMan https://github.com/BrikerMan I agree with you now. The proposal I drew this afternoon appears to be too ideal and I'm afraid it won't ever come true in that simple way.

Yes, now the only thing we should discuss is where to use the package name kashgari-tf.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKSOWMGKXSWLFZEPCH3P4SMV3A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYWYMRA#issuecomment-506299972, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKV446E5TXGFQKBCXETP4SMV3ANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

@BrikerMan I have fix a bug in #108, but that part of the code was modified in 9e72bbb. Then the problem reappears.

reproduce:

import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)

Could you help me fix this bug again, I have no access to multi-gpu device now. I can't reproduce this issue.

BrikerMan commented 5 years ago

@alexwwang Maybe we just release tf.keras version with less classification model, Then consider how to add more models.

haoyuhu commented 5 years ago

@BrikerMan I have fix a bug in #108, but that part of the code was modified in 9e72bbb. Then the problem reappears.

reproduce:
import kashgari
from kashgari.corpus import ChineseDailyNerCorpus
from kashgari.tasks.labeling import BiLSTM_Model

train_x, train_y = ChineseDailyNerCorpus.load_data('train', shuffle=False)
model = BiLSTM_Model()

model.build_model(train_x, train_y)

model = kashgari.utils.convert_to_multi_gpu_model(model, gpus=2)
model.fit(train_x, train_y)
Could you help me fix this bug again, I have no access to multi-gpu device now. I can't reproduce this issue.

I prefer to add a member function BaseModel.build_multi_gpu_model(...) instead of current utils.convert_to_multi_gpu_model(...). This will make it easier to solve this problem.

alexwwang commented 5 years ago

@alexwwang Maybe we just release tf.keras version with less classification model, Then consider how to add more models.

@BrikerMan I may pass the test for those models refactored today. Then the new edition could be launched, I think. Pls remember to help me clarify the doubtness I met during the #118 work.

BrikerMan commented 5 years ago

problem

Ok, if so let's add BaseModel.build_multi_gpu_model(...) and BaseModel.build_tpu_model(...)

BrikerMan commented 5 years ago

@alexwwang will do, one more thing, customize layer issue may be related to tensorflow version. My CRF layer works fine at first, but now it will crash #120 . I need to fix this. Let's work on tensorflow==1.14.0

alexwwang commented 5 years ago

@BrikerMan Yes, under tf1.14 and I've solved the customize layer issues related yesterday. They're related to type conversion and difference of api between tf.keras and Keras. Maybe we met different situations.

BrikerMan commented 5 years ago

My CRF layer work on 1.13.1, but will crash at 1.14... Let's specify a tf-version for every Kashgari releases. Or we will have to deal with endless problems. This version lets work on 1.14.0.

alexwwang commented 5 years ago

I decide to make a qr first, and you will know where the problem may lay on. Are you sure all the weights you add in CRF without setting trainable?

BrikerMan commented 5 years ago

@alexwwang Here is still 4 error with classification models, please check it out.

alexwwang commented 5 years ago

@BrikerMan where could I see the error info?

BrikerMan commented 5 years ago

@BrikerMan where could I see the error info?

https://travis-ci.com/BrikerMan/Kashgari

alexwwang commented 5 years ago

Thanks.

Yes, an error occurred when refactor the struct codes. And it has been corrected.

I'll make a pr right now.

BrikerMan commented 5 years ago

@alexwwang @HaoyuHu All issue fixed, test-case passed. Now it is time to finally finalize this release. Here is the new todo list.

[ ] Clean up the code
[ ] Add classification tutorial
[ ] Add embedding tutorial
[ ] choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use kashgari-tf, we need to remind users to migrate to kashgari-tf.

alexwwang commented 5 years ago

Or a whole new name, say, shouwen, named as a Chinese monk in 9th century?

On Fri, 28 Jun 2019 at 22:36, Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu https://github.com/HaoyuHu All issue fixed, test-case passed. Now it is time to finally finalize this release. Here is the new todo list.

Add convert model to multi-gpu/TPU function to base-model class

Clean up the code

Add classification tutorial

Add embedding tutorial

choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use kashgari-tf, we need to remind users to migrate to kashgari-tf.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKTMUWMZ5M3I52YGKILP4YOYHA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2IBWQ#issuecomment-506757338, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKQA2OZJ3QDJSNJQSNLP4YOYHANCNFSM4HIAZOGQ .

Qian16 commented 5 years ago

@alexwwang @HaoyuHu All issue fixed, test-case passed. Now it is time to finally finalize this release. Here is the new todo list.

[ ] Clean up the code

[ ] Add classification tutorial

[ ] Add embedding tutorial

[ ] choose pip package name, kashgari or kashgari-tf

If we use kashgari, it will break the existing code. But if we use kashgari-tf, we need to remind users to migrate to kashgari-tf.

Nice job guys. Don't forget also to add tutorials about multilabel and multioutput. Thanks.

BrikerMan commented 5 years ago

@alexwwang I prefer stick to the name Kashgari.

alexwwang commented 5 years ago

Well then, -tf suffix may be better.

On Sat, 29 Jun 2019 at 10:17, Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang I prefer stick to the name Kashgari.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXFO35HFXL6264GDF3P43A4ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3PQYA#issuecomment-506919008, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKVZ2FWO7D2SU22B3YDP43A4ZANCNFSM4HIAZOGQ .

alexwwang commented 5 years ago

But don't you think it's a little long? How about kash-tf?

On Sat, 29 Jun 2019 at 11:50, Alex Wang azure.ww@gmail.com wrote:

Well then, -tf suffix may be better.

On Sat, 29 Jun 2019 at 10:17, Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang I prefer stick to the name Kashgari.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXFO35HFXL6264GDF3P43A4ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3PQYA#issuecomment-506919008, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKVZ2FWO7D2SU22B3YDP43A4ZANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

@alexwwang We could use kashgari-tf as package name. But still use import Kashgari in the code. Just like tensorflow-gpu.

BrikerMan commented 5 years ago

@alexwwang @HaoyuHu Guys. Do you have any time this weekend to work on the document. I am planning to release the first tf.keras version this weekend. With kashgari-tf package name with 0.5.0 version.

alexwwang commented 5 years ago

Sorry but I could only have time working on it on week days.

And I have to check the program carefully to make it clear how to use it, especially on stacked-embedding, hyper parameter settings, the optimizor part, and model save with customized layers. These parts are not covered by unit test yet.

On Sat, 29 Jun 2019 at 11:59, Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang @HaoyuHu https://github.com/HaoyuHu Guys. Do you have any time this weekend to work on the document. I am planning to release the first tf.keras version this weekend. With kashgari-tf package name with 0.5.0 version.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKQPNO2643K72DLAW3TP43M3LA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3QYNQ#issuecomment-506924086, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKTR2YCI75SARO5CQ23P43M3LANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

Sorry but I could only have time working on it on week days.

And I have to check the program carefully to make it clear how to use it, especially on stacked-embedding, hyper parameter settings, the optimizor part, and model save with customized layers. These parts are not covered by unit test yet.

Yes. I am a little bit in rush. Let’s cool down. I will finish embedding tutorial this weekend. Take your time, I will try add more test case too. Let’s release a little bit later.

alexwwang commented 5 years ago

Thank you for your understanding. I plan to implement the DPCNN model next week. I think it's interesting and do fast and well in certain classification tasks.

On Sat, 29 Jun 2019 at 20:36, Eliyar Eziz notifications@github.com wrote:

Sorry but I could only have time working on it on week days.

And I have to check the program carefully to make it clear how to use it, especially on stacked-embedding, hyper parameter settings, the optimizor part, and model save with customized layers. These parts are not covered by unit test yet.

Yes. I am a little bit in rush. Let’s cool down. I will finish embedding tutorial this weekend. Take your time, I will try add more test case too. Let’s release a little bit later.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKS7XGDB3O4T4KLLVRTP45JK7A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3YA4Y#issuecomment-506953843, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKSF2VTPVBH7KBKUFULP45JK7ANCNFSM4HIAZOGQ .

haoyuhu commented 5 years ago

@alexwwang @HaoyuHu Guys. Do you have any time this weekend to work on the document. I am planning to release the first tf.keras version this weekend. With kashgari-tf package name with 0.5.0 version.

I am afraid that I don't have a lot of time to work on the document due to my busy work. I am so sorry. 😿 BTW, I am very much looking forward to the release of the tf.keras version.

haoyuhu commented 5 years ago

Thank you for your understanding. I plan to implement the DPCNN model next week. I think it's interesting and do fast and well in certain classification tasks. … On Sat, 29 Jun 2019 at 20:36, Eliyar Eziz @.***> wrote: Sorry but I could only have time working on it on week days. And I have to check the program carefully to make it clear how to use it, especially on stacked-embedding, hyper parameter settings, the optimizor part, and model save with customized layers. These parts are not covered by unit test yet. Yes. I am a little bit in rush. Let’s cool down. I will finish embedding tutorial this weekend. Take your time, I will try add more test case too. Let’s release a little bit later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77?email_source=notifications&email_token=AAGRFKS7XGDB3O4T4KLLVRTP45JK7A5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3YA4Y#issuecomment-506953843>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKSF2VTPVBH7KBKUFULP45JK7ANCNFSM4HIAZOGQ .

I am also studying DPCNN recently. This is very nice if we can use DPCNN in a future release. That's sweet of you!

BrikerMan commented 5 years ago

Anything else needs to check? finished our first checklist. Going to finalize the 0.5.0 release.

[x] Clean up the code
[x] Add classification tutorial
[x] Add embedding tutorial
[x] choose pip package name, kashgari-tf

BrikerMan commented 5 years ago

@alexwwang Maybe we just get rid of those codes, since we have already broken our user's code and they need to start with the whole new version.

BLSTMModel = BiLSTM_Model
BGRUModel = BiGRU_Model
CNNModel = CNN_Model
... ...

BrikerMan commented 5 years ago

🎉🎉🎉🎉 Released 0.5.0 🎉🎉🎉🎉

alexwwang commented 5 years ago

Any idea to support some short alias for those model names? I think that would be friendly to users.

On Thu, Jul 11, 2019, 13:33 Eliyar Eziz notifications@github.com wrote:

@alexwwang https://github.com/alexwwang Maybe we just get rid of those codes, since we have already broken our user's code and they need to start with the whole new version.

BLSTMModel = BiLSTM_Model BGRUModel = BiGRU_Model CNNModel = CNN_Model... ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKXRWGUVRYXEZHD6GLDP63A2ZA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZVRW4Y#issuecomment-510335859, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKUXWVLK34CIYL6AMGTP63A2ZANCNFSM4HIAZOGQ .

BrikerMan commented 5 years ago

Any idea to support some short alias for those model names? I think that would be friendly to users.

Sounds good, any specific idea? Maybe we should open a new issue for this matter.

alexwwang commented 5 years ago

Ok. Let me see how to abridge them.

On Sat, Jul 13, 2019 at 16:57 Eliyar Eziz notifications@github.com wrote:

Any idea to support some short alias for those model names? I think that would be friendly to users.

Sounds good, any specific idea? Maybe we should open a new issue for this matter.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/BrikerMan/Kashgari/issues/77?email_source=notifications&email_token=AAGRFKRXYLO4QRNPWR5OYQLP7GKGDA5CNFSM4HIAZOG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3NKAY#issuecomment-511104259, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRFKUGLCXRDJ3S7ZQ4NCTP7GKGDANCNFSM4HIAZOGQ .

BrikerMan / Kashgari