search

francescamen / SHARP

Learning algorithm for human activity recognition with a commercial IEEE 802.11ac router
GNU General Public License v3.0
31 stars 15 forks source link

Results don't match the provided reference #2

Open Funhador opened 1 year ago

Funhador commented 1 year ago

Dear @francescamen,

Thanks for the work carried out. I have been having problems validating your article's result and I was wondering if you could give me some insights.

After training the network, testing it with any other dataset except the one generated from the same set of Doppler traces used for training results in highly different accuracies and F1-scores from the ones shown in the paper. However, a warning from the scikit-learn is raised for undefined metrics (as shown in the attached image). As this warning doesn't appear in the exception I mentioned, I believe the problem is related to this but until now I've been fruitless in finding its root. UndefinedMetricWarning

francescamen commented 1 year ago

Dear @Funhador,

I rechecked my code and dataset and I was able to replicate the results of the paper. I created a shared folder where I uploaded the pre-computed Doppler traces, you can find the link in the repository README at https://github.com/francescamen/SHARP/tree/main#pre-computed-doppler-traces In the same folder, I uploaded some plots of Doppler traces and some confusion matrices obtained through the pre-trained model.

I also uploaded a new file in this repository to visualize the Doppler traces, please see https://github.com/francescamen/SHARP#doppler-computation.

Can you please try to run the code with those pre-computed Doppler traces to see whether you can replicate the results?

Thank you for your feedback! Francesca

riverleejames commented 3 months ago

Subject: Issue Replicating Paper Results with Pre-computed Doppler Traces

Dear @francescamen ,

Like @Funhador , I encountered difficulties in replicating the results presented in your paper, despite using the pre-computed Doppler traces you provided. Wondering if this might be due to library updates, I meticulously set up a Docker container with the exact library versions specified in your instructions. Unfortunately, I still ended up with the same outcomes.

Please see results below: cm_complete_different_E,L,W,R,J_S7a_band_80_subband_1.pdf

cm_complete_different_E,L,W,R,J_S7a_band_80_subband_1_max_merge.pdf

Compared to the results you provided: cm_complete_different_E,L,W,R,J_S7a_band_80_subband_1.pdf cm_complete_different_E,L,W,R,J_S7a_band_80_subband_1_max_merge.pdf

Thank you,

River.