search

openvinotoolkit / openvino_notebooks

📚 Jupyter notebook tutorials for OpenVINO™
Apache License 2.0
2.29k stars 791 forks source link

Models for the Google Summer of Code 2023 prerequisite task #832

Closed adrianboguszewski closed 9 months ago

adrianboguszewski commented 1 year ago

The models listed below are for the GSoC 2023 prerequisite task only.

We provide several potential candidates. Please select only one which hasn't already been selected (look at the checkboxes and comments below). When you decide, assign a model to you by adding a comment with the model name. Then we will tick it to mark reserved.

If you struggle, you can reassign yourself to another non-taken model. However, we can do it only once.

When you create a PR, please follow the self-checklist below:

Object detection:

Rotated object detection:

Semantic Segmentation:

Instance Segmentation:

Action/Gesture recognition:

Background matting:

Old Photos Restoration/Image colorization/Image denoising/super resolution:

Depth estimation:

Text classification:

Token classification:

Text generation:

Text Summarization

Question Answering

Sound classification:

sahilpmehra commented 1 year ago

Hi, could you please assign me Rotated FCOS? Thanks!

Matrixmang0 commented 1 year ago

Hello everyone!

I would love to work on YOWOv2. If still unreserved, I request the admin to please reserve it for me.

Thank you

hadyy17 commented 1 year ago

Hello, as i have assigned BeIT to myself and i am currently working on developing a notebook. I has one question, for the pre-requisite task, do i need to modify the pre-trained model mentioned with certain usecase for openvino support or i can try making a model for my own usecase using BeIT and then add required support?

Any help is much apreciated.

adrianboguszewski commented 1 year ago

Could you assign CenterNet to me? Thanks you

@18yz153, unfortunately, CenterNet model has been chosen a few comments above you. Please select another one.

paxF3E commented 1 year ago

Hello, I shall be willing to work with Sound Classification:speech emotions recognition (wav2vec). Can I please have this model assigned to me? Thank you.

eaidova commented 1 year ago

Hello, as i have assigned BeIT to myself and i am currently working on developing a notebook. I has one question, for the pre-requisite task, do i need to modify the pre-trained model mentioned with certain usecase for openvino support or i can try making a model for my own usecase using BeIT and then add required support?

Any help is much apreciated.

@hadyy17 what do you mean under own use case? The most important thing that model should solve semantic segmentation task and running with openvino. If you can demonstrate own use case with preserving these conditions, it is fine. Generally, using model from link provided with the task should be enough. (it contains ADE20k trained model from HuggingFace hub)

matrixbot123 commented 1 year ago

Hi everyone ! I would like to work on DistilBART . Thank You

AlexFierro9 commented 1 year ago

Hi, I had assigned TimesFormer for myself, but I'm not able to convert the model to onnx correctly (some error with pytorch on my hardware), I've posted my issue on pytorch discuss but haven't gotten a response, can I change my model to a Tensorflow Based, if yes then please assign SSD MobileNet V2!

Akshit17 commented 1 year ago

Hello everyone! I would like to work on MiniLM. Can I have this model assigned to me? Thank you.

AniketARS commented 1 year ago

Hello Everyone, I would like to work on SETR - (https://github.com/fudan-zvg/SETR) model for my GSoc Application. I'm guessing it is unreserved, if anyone already working on this kindly inform me i will change my choice

Thanks, Aniket

Paulooh007 commented 1 year ago

Hi, Everyone. I would like to assign FasterRCNN Inception ResNet v2 to myself

fajemila commented 1 year ago

I would like to assign DistilBert to myself

Tatwansh commented 1 year ago

I would like to assign DETR to myself

SandeepaDevin commented 1 year ago

I would like to assign YOLOS to myself

18yz153 commented 1 year ago

I would like to assign YOLOR to myself

thegeek13242 commented 1 year ago

I would like to assign YoloF to myself.

nischay7 commented 1 year ago

Hi, I would like to assign ReDet to myself

zhumakhan commented 1 year ago

OPT Hi, I would like to assign OPT to myself.

Thanks

wulongjian commented 1 year ago

Hi, I would like to assign Roi_trans to myself. Thanks.

AlexFierro9 commented 1 year ago

Hi, There is an already built copy of SSD NET V2 at here at TF hub, and while the model can be rebuilt again from thesecheckpoints, available at TF model zoo too, This way also requires using TensorFlow Object Detection API that needs to be separately installed, to be able to recompile the model and save it. So should I use the already existing saved model or build it again?

eaidova commented 1 year ago

@AlexFierro9 you can use any of these modesl, but please pay attention that they are 2 different models, the first one in Tf Object detection API named as SSD MobileNet V2 FPNLite 320x320

hazrulakmal commented 1 year ago

Hi, this might seem like a dumb question. Does one need a machine with an Intel processor to meaningfully contribute to this organization and complete the prerequisite task with OpenVINO toolkits? Would someone with, say, a Ryzen processor be able to complete the task?

eaidova commented 1 year ago

@hazrulakmal unfortunately, there is no openvino prebuild packages for AMD processors, we can recommend using google colab or any other online jupyter notebooks environment to complete the prerequisite task

mhy-666 commented 1 year ago

Hi, can I assign Deberta to myself? Thanks.

pyther-hub commented 1 year ago

@eaidova sorry for the inconvenience but assign me NanoDet as there is no active documentation for yolo v7 face

sususama commented 1 year ago

Hi, I want to assign Photo2Cartoon.Thanks.

ryan-utopia commented 1 year ago

Hi, could you please assign me UltraFace? If I work on this successfully before March 20th, I will also try DAMO YOLO, Thanks!

Nouran-Muhammad commented 1 year ago

Hi, Could you please assign me ESPNet? Thank you

weronikazak commented 1 year ago

Hello, could you please assign me Coltran? Thank you!

wulongjian commented 1 year ago

Hello, I had assigned Roi_trans for myself, but I'm not able to convert it correctly in intel cpu platform. As far as I konw, this model only support nvidia inference currently(see this issue#551 and get_started).
Can I change my model to others? if yes then please assign Robust Video Background matting to me, thanks a lot.

AlexFierro9 commented 1 year ago

Hi, So I had a doubt with model conversion, I've been referring to the model conversion guide here Converting TensorFlow Object Detection API Models as according to the documentation and Colab tutorials available here. The mo command for conversion requires me to also specify --transformations_config for argument, unfortunately this model was not downloaded from tensorflow object detection api zoo, so I have no clue which file to use here, should I outright skip this parameter or should I use the json file specified for ssd_v2? btw my model is ssd_mobilenet_v2_320x320_coco17_tpu-8

shrey-2803 commented 1 year ago

Hi, I would like to assign FinBERT to myself. Thank You!

sharvesh642 commented 1 year ago

Hi @adrianboguszewski @eaidova , I was assigned with the movinet task and while I was trying to load the tensorflow model after converting it into OPENVINO IR I encountered an error - "Model file model/v3-small_224_1.0_float.xml cannot be opened!" When I was trying to Load the model in OpenVINO Runtime with ie.read_model I encountered the error - "Model file model/MiDaS_small.xml cannot be opened!". Am I doing something wrong how can I rectify it?

aadhamm commented 1 year ago

Hi, @adrianboguszewski @eaidova

I would like to assign "Real-ESRGAN (for real images)" to myself. Thank you for this insightful list and best of luck for all of us.

eaidova commented 1 year ago

@sharvesh642 which platform do you use for development? Please also check that your path to model is correct and both IR files (XML and BIN )are located in the same directory and have the same name without extension

eaidova commented 1 year ago

Hi, So I had a doubt with model conversion, I've been referring to the model conversion guide here Converting TensorFlow Object Detection API Models as according to the documentation and Colab tutorials available here. The mo command for conversion requires me to also specify --transformations_config for argument, unfortunately this model was not downloaded from tensorflow object detection api zoo, so I have no clue which file to use here, should I outright skip this parameter or should I use the json file specified for ssd_v2? btw my model is ssd_mobilenet_v2_320x320_coco17_tpu-8

@AlexFierro9 you can use any suitable SSD mobilenet v2 model which you can run and convert, so model selection should not be a problem, right? :)

TF object detection models saved using in saved model format correspond to TF2.0 and should work without pipeline config

pyther-hub commented 1 year ago

@eaidova My model is nanodet in that when I compile my model 

from openvino.runtime import Core
core = Core()
model = core.read_model('/content/nanodet_model.xml')
compiled_model = core.compile_model(model, 'CPU')

after this 

output_blob = compiled_model.output(0)
predictions = torch.from_numpy(compiled_model(input_tensor)[output_blob])

the shape of my predictions is 2125, 112 where 2125 is number of boxes now the main problem comes up of 112, ideally it should be 85 1+4+80 (one for probability of an object, 4 is the coordinate of the box, 80 is the prob. of each class)

after doing a ton of dugging I found out that the model returns this only after which it needs to be passed through post_process where it decodes the bboxes and all the functions are complex and inter linked I am not able to move from here like how can I get the required output from this 112

MR-ENVYR commented 1 year ago

Hello, My model is distilroberta. There is no tiny model for it so should I just use the model file available and write an export script for the model? I had a doubt as the example notebook used an export script which was available in the github repo of the model. So should I follow a similar approach of the export script or can I use any approach as convenient? Thanks in advance :D

eaidova commented 1 year ago

Hello, My model is distilroberta. There is no tiny model for it so should I just use the model file available and write an export script for the model? I had a doubt as the example notebook used an export script which was available in the github repo of the model. So should I follow a similar approach of the export script or can I use any approach as convenient? Thanks in advance :D

you can use any approach which can be suitable

eaidova commented 1 year ago

@eaidova My model is nanodet in that when I compile my model

from openvino.runtime import Core
core = Core()
model = core.read_model('/content/nanodet_model.xml')
compiled_model = core.compile_model(model, 'CPU')

after this

output_blob = compiled_model.output(0)
predictions = torch.from_numpy(compiled_model(input_tensor)[output_blob])

the shape of my predictions is 2125, 112 where 2125 is number of boxes now the main problem comes up of 112, ideally it should be 85 1+4+80 (one for probability of an object, 4 is the coordinate of the box, 80 is the prob. of each class)

after doing a ton of dugging I found out that the model returns this only after which it needs to be passed through post_process where it decodes the bboxes and all the functions are complex and inter linked I am not able to move from here like how can I get the required output from this 112

@pyther-hub Possibly, I do not quite understand what the issue is, what is the problem to apply post-processing to model output after openvino inference? It is normal practice that some models can require additional steps for getting results. e.g. if you look on yolov7 tutorial it is also requires to make non-maximum suppression for boxes after inference

mhy-666 commented 1 year ago

Hi, can I assign ELECTRA to myself? Thanks.

pyther-hub commented 1 year ago

@eaidova My model is nanodet in that when I compile my model

from openvino.runtime import Core
core = Core()
model = core.read_model('/content/nanodet_model.xml')
compiled_model = core.compile_model(model, 'CPU')

after this

output_blob = compiled_model.output(0)
predictions = torch.from_numpy(compiled_model(input_tensor)[output_blob])

the shape of my predictions is 2125, 112 where 2125 is number of boxes now the main problem comes up of 112, ideally it should be 85 1+4+80 (one for probability of an object, 4 is the coordinate of the box, 80 is the prob. of each class) after doing a ton of dugging I found out that the model returns this only after which it needs to be passed through post_process where it decodes the bboxes and all the functions are complex and inter linked I am not able to move from here like how can I get the required output from this 112

@pyther-hub Possibly, I do not quite understand what the issue is, what is the problem to apply post-processing to model output after openvino inference? It is normal practice that some models can require additional steps for getting results. e.g. if you look on yolov7 tutorial it is also requires to make non-maximum suppression for boxes after inference

ma'am the issue is the object returns a diff. output and it needs to be post processed but the post processing step is very very complex, I am trying to. implement that I have one more backup option which I will look later I would keep you posted regarding my progress, thank you for your response

pyther-hub commented 1 year ago

@eaidova I would like to change my model from nanodet to Colorizer I apologize for any inconvenience I may have caused. Despite my best efforts, I encountered difficulties in resolving the output issue. I reached out to several contributors via email, but one of them informed me that the code required for this conversion would be quite challenging to write. Moreover, the output I obtained did not match the model.forward result, which ultimately led me to a dead end.

mhy-666 commented 1 year ago

Hi, can I assign ELECTRA to myself? Thanks.

I'm sorry, I didn't notice the requirement that only one task can be assigned to a person. I have already completed the task that I previously received.

adrianboguszewski commented 1 year ago

@eaidova I would like to change my model from nanodet to Colorizer I apologize for any inconvenience I may have caused. Despite my best efforts, I encountered difficulties in resolving the output issue. I reached out to several contributors via email, but one of them informed me that the code required for this conversion would be quite challenging to write. Moreover, the output I obtained did not match the model.forward result, which ultimately led me to a dead end.

It's ok. Reassigned.

adrianboguszewski commented 1 year ago

Hi, can I assign ELECTRA to myself? Thanks.

I'm sorry, I didn't notice the requirement that only one task can be assigned to a person. I have already completed the task that I previously received.

Yes. One notebook implementation is enough. Please be sure you will have opened a PR (it's part of the task) with your work by 4th April.

pyther-hub commented 1 year ago

@adrianboguszewski sir I have made the notebook for coloriser task but while doing documentation I found that one [implementation of it is already there] (https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/222-vision-image-colorization) , what shall I do now?

adrianboguszewski commented 1 year ago

@adrianboguszewski sir I have made the notebook for coloriser task but while doing documentation I found that one [implementation of it is already there] (https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/222-vision-image-colorization) , what shall I do now?

Just create a PR. If it was listed above, it means we would like to have it anyway. When you create a PR, we will let you know about the next steps.

lucifertrj commented 1 year ago

Hi @adrianboguszewski ,

Please assign YoloV7 Face to me

AlexFierro9 commented 1 year ago

923