search

soldair / node-walkdir

Walk a directory tree emitting events based on the contents. API compatable with node-findit. Walk a tree of any depth. Fast! Handles permission errors. Stoppable. windows support. Pull requests are awesome. watchers are appreciated.
MIT License
130 stars 22 forks source link

catch less path than fs.readdir #33

Closed bi-kai closed 6 years ago

bi-kai commented 6 years ago

windows properties

112,033 Files, 14,187 Folders

with fs.readdirSync

file:112033; dir:14187; total:126220 time:22.163s

with walkdir

callback: async total:126128 sync total:126134 time:24.664s

emitter: file:111947; dir:14181; total:126128 time:13.454s

compare

walkdir sync less than fs.readdirSync: missed dir:9; missed file:77

walkdir got less file than origin api, have you ever noticed this issue? sync func get more files than async, why is this difference?

bi-kai commented 6 years ago

by compare, these file are no count in walkdir sync: walkdir sync less than fs.readdirSync: missed dir:9; missed file:77

file no found: D:\xxx\mysql\share\japanese\errmsg.sys
file no found: D:\xxx\xxx\3rdparty\lib\tcllib\win32\tcl\lib\tcl8.4\http2.5\pkgIndex.tcl
file no found: D:\xxx\xxx\3rdparty\nodejs\node_modules\socket.io\package.json
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\bme.common\theme\default\images\page_btn.png
dir no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\arrow_d.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\icon.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\inp_1.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_btn.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_btn_lv-g.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_btn_lv.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_btn_on-g.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_btn_on.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_l.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_left_disabled.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_r.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\page_right_disabled.png
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\pagination\images\pagination\temp_page.png
dir no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\progressBar\images\progressBar
file no found: D:\xxx\xxx\framework\com.iemp.framework\ui\com.huawei.oms.framework.homepage.web-2.0-SNAPSHOT\themes\default\components\eview\css\eview\widgets\progressBar\images\progressBar\progressbar.png
file no found: D:\xxx\xxx\sysagent\etc\sysconf\agent_win\med_agent.xml
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\progressBar\images\progressBar
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\progressBar\images\progressBar\progressbar.png
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\spinner\images\spinner
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\spinner\images\spinner\numricstepper_bar.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\spinner\images\spinner\time_down.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\spinner\images\spinner\time_up.png
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\header.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\hover_left_arrow.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\hover_right_arrow.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\left_arrow.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\next_grey.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\prev_grey.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\right_arrow.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\selected.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\components\eview\css\eview\widgets\timeline\images\timeline\unselected.png
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets
dir no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\downbutton.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\downbutton.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\downbuttonbright.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\downbuttonbright.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\forward_enabled_hover.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\last.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\minus.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\next.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\plus.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\previous.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scroll.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scroll.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scrollbright.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scrollbright.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scrollpane.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\scrollpane.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\sort_asc.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\sort_asc_disabled.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\sort_both.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\sort_desc.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\sort_desc_disabled.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\tree_arrows.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\tree_expcol.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\tree_file.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\tree_folder.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\tree_folder_open.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\upbutton.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\upbutton.png
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\upbuttonbright.gif
file no found: D:\xxx\xxx\themes\iDEO\iDEOSkin\framework\eview_modules\portal\com.eview.component\1.0.0\components\table\images\eview\widgets\pagepane\upbuttonbright.png
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\PDSN9660\PDSN9660 V900R007C02\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\PDSN9660\PDSN9660 V900R007C06\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\UGW9811\UGW9811 V900R001C05\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\UGW9811\UGW9811 V900R011C01\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\UGW9811\UGW9811 V900R011C80\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\CHN\ini\Alarmcfg\UGW9811\UGW9811 V900R012C00\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\PDSN9660\PDSN9660 V900R007C02\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\PDSN9660\PDSN9660 V900R007C06\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R001C05\ealmorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R010C00\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R010C02\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R010C04\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R011C01\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\UGW9811\UGW9811 V900R011C80\almorig.cfg
file no found: D:\xxx\xxx\tools\Security Log And Softpara Browse\Security Log And Softpara Browse Ver1.1.2\ENG\ini\Alarmcfg\WASN9770\WASN9770 V300R003C02\almorig.cfg

I wander, have walkdir filtered some folder name and file extensions?

soldair commented 6 years ago

it shouldn't. It does ignore files that are simply links to other files its already seen though. if you get closer to figuring it out i would be happy to make walkdir better

bi-kai commented 6 years ago

I have tried many times, walkdir did missed some files. If you try to iterator more than 100k files, you may find this issue. I post a summarize here: https://cnodejs.org/topic/5b34d77c5c8f1ea723a56ac5#5b440cd135342ab0690613c2