Fourdee
commented
6 years ago @MichaIng
Ah @Fourdee we should remove hostname for anonymity? Can't think of value we get through this.
Yep 👍 , we can remove it, no benefit to us knowing that information.
Closed MichaIng closed 6 years ago
Fourdee
commented
6 years ago @MichaIng
Ah @Fourdee we should remove hostname for anonymity? Can't think of value we get through this.
Yep 👍 , we can remove it, no benefit to us knowing that information.
MichaIng
commented
6 years ago I played around a bid and created a simple result page: https://dietpi.com/survey/
Ugly I know, and there is still a glitch with some software titles (bottom) being somehow broken over two lines. Ah now that I write it, I think I know what's the issue about it: sed -i 's/ /_/g' "$file
€: Solved!
It is updates periodically every 15 minutes. Hmm I will add some UTC time stamp as well.
NB:
dietpi-survey menu. As the public upload user has no read nor file list permissions, it is not possible to remove the file entirely. However it is interesting as well to know how much users requested to clear their data to check acceptance (see result page). But again, the files itself are empty (0 bytes) then.I will go and upload the result page creation script to a separate GitHub branch, in case anybody wan't to make it more beautiful 🙂.
Fourdee
commented
6 years ago @MichaIng
Love this, great work 👍
Was having a play with 'Opted out':
if [[ ! -s $file ]]; then
((SURVEY_COUNT_EMPTY++))
device_name='Unknown_(Opted_Out)'
((aDEVICE_NAME[$device_name]++))
continue
fiDo you think we need it?
MichaIng
commented
6 years ago @Fourdee Not sure, we would need to add dummy names for all infos then, for consistency?
Currently, summarizing the counts of each info type matches Available unique survey files.
Purged unique survey files gives the info about how much users actively opted out and wanted to have their data purged in case. Not sure if it makes sense to add this number to each category again as unknown 😉. Maybe a better naming could make things clearer again.
But still we do not have the total users count, as if users just opt out without purge request, we have no file at all.
Hmm actually would be nice to have the overall count as well, maybe opt out users just by default always send empty files? Otherwise if I don't want to send data, I also would not want to have empty files sent, I would want to have the whole process skipped.
Fourdee
commented
6 years ago @MichaIng
Hmm actually would be nice to have the overall count as well, maybe opt out users just by default always send empty files? Otherwise if I don't want to send data, I also would not want to have empty files sent, I would want to have the whole process skipped.
I'd like to keep the initial send, even if the user opts out and its cleared. Allows us to at least have a total count of unique installs from no data.
But still we do not have the total users count, as if users just opt out without purge request, we have no file at all.
Good point, maybe would could simply remove '1' 'No, I want to opt out DietPi-Survey.'?
Hmmm, i'll have a play around with script, see what I can come up with.
Fourdee
commented
6 years ago @MichaIng
Not sure if it makes sense to add this number to each category again as unknown 😉. Maybe a better naming could make things clearer again.
Yeah, just had the idea of making each table add up to the total. May even make users change their mind to OPT IN lol (we can but hope).
Fourdee
commented
6 years ago Fourdee
commented
6 years ago @MichaIng
Ok, the way I want this is:
At the very least, we have a right to know how many unique installs of DietPi there is.
Commit sent and tested.
Its a shame, the companies who abuse peoples data (Facebook, MS, etc) and sell it for profit, have destroy it for the rest of us who have no ill agenda.
MichaIng
commented
6 years ago @Fourdee
€: Added timestamp to survey page. Hmm it already quite takes a while until finished now. Maybe we need to do some performance improvement with growing survey files number. Copy and handle everything in ram, or make the upload folder a ramdisk directly from where re regularly run backups? Or indeed storing everything in a database or global array that is just updated by new sent data:
Fourdee
commented
6 years ago @MichaIng
Yep, see what you mean, getting a little lengthy to process:
root@DietPi:~# time /etc/cron.minutely/survey-report
real 0m48.765s
user 0m2.152s
sys 0m3.872sRAM should be fine.
Thinking we need to reduce the pipe's, awk and sed's, might speed it up?
Another option is we could change the survey formatting, make them all variables and simply load the file? eg:
#-------------------------
dietpi_survey_version[$uuid]=5
#-------------------------
upload_count[$uuid]=1
dietpi_version_core[$uuid]=6
#etcWe should aim to do the minimal processing client side, take load off the server.
MichaIng
commented
6 years ago @Fourdee ~ 2000 unique installations on v6.9+ in ~6 days 👍.
I will go through the code for now and check if I can reduce the pipes. Of course larger update intervals also reduce load. 15 minutes was interesting to follow update counts the first hours and days, but now hourly or even larger intervals would be enough.
Yeah for next version making the survey files source-able is a good idea. Still the arrays would need to be converted. Hmm we could even already place the summation inside:
#-------------------------
((${dietpi_survey_version[5]}++))
#-------------------------
((${upload_count[1]}++))
((${dietpi_version_core[6]}++))
#etc((${array[$index]:-1}++))?Fourdee
commented
6 years ago @MichaIng
I was thinking something along the lines of:
#!/bin/bash
index_input=\$1
dietpi_survey_version[\$index_input]=5
upload_count[\$index_input]=$uploadcount
g_hw_model[\$index_input]=2Then we could simply do the following in the server script:
local index=0
for file in /home/dietpi-survey/survey/*.txt
do
. $file $index
((index++))
doneThen all the arrays are loaded. The issue is with installed software, might need to covert it to a 2d array on the client before sending, eg:
root@DietPi:~# grep '=2' /DietPi/dietpi/.installed | sed 's/E\[/E\[$index_input\]\[/'
aSOFTWARE_INSTALL_STATE[\$index_input][0]=2
aSOFTWARE_INSTALL_STATE[\$index_input][103]=2
aSOFTWARE_INSTALL_STATE[\$index_input][105]=2Or we could create a new input for dietpi-software list_survey and print only the following:
openssh client
dietpi-ramlogor, the following then convert as needed during sourcing on server
root@DietPi:~# dietpi-software list | grep ' =2 ' | awk '{print $2}' | sed 's/^/software_index_installed=/' software_index_installed=0
software_index_installed=103
software_index_installed=105@Fourdee
To have the files source-able should already speed up creation indeed. But as said with:
g_hw_model[\$index_input]=2
if we want to have as now a count of how many users user model 2, we would still need an array conversion like
for i in ${g_hw_model[@]}; do ((aHW_MODEL[$i]++))
To have ((aHW_MODEL[$G_HW_MODEL_NAME]++)) directly within the survey files, would allow us to skip this conversion as well and does not have any effect on client side creation speed.
Btw the index does not need to exist to allow ++. If it doesn't exist, it will be created with value 1. See current survey_report script, where I removed the initial index creation.
About the dietpi-software list usage: I already thought about the same, but the list creation probably takes too much time and could cause issues on first run updates and within dietpi-software calling dietpi-software itself again? I think the survey file creation process should be as fast and clean as possible to make users not opt out because of annoying delay and terminal outputs.
The survay_report script could have one final conversion for software list:
for i in ${!aSOFTWARE_INDEX[@]}; do aSOFTWARE[${aSOFTWARE_NAME[$i]}]=${aSOFTWARE_INDEX[$i]}; done}
MichaIng
commented
6 years ago @Fourdee
Replaced sed and awk by bash internal variable manipulation ${VAR##* : } etc: http://wiki.bash-hackers.org/syntax/pe
root@DietPi:~# time /etc/cron.minutely/survey-report
real 0m37.277s
user 0m3.428s
sys 0m4.516s😃
Fourdee
commented
6 years ago @MichaIng
RAM test, I believe its hitting swapfile.
root@DietPi:~# time /etc/cron.minutely/survey-report_4D_testing
real 0m35.267s
user 0m3.196s
sys 0m4.244s~Hmm, values dont add up:~
Me = Idiot 🤣. Was seeing "Users opted in:" as total. More coffee required.
In = 1366
Users opted out + purge: | 1155
Dropbear = 816#Purged
root@DietPi:~# ls -lha /home/dietpi-survey/survey/ | grep ' 0 ' | wc -l
1155
root@DietPi:~# cat /home/dietpi-survey/survey/*.txt | grep '\[104\]' | wc -l
819 #?? Max of possible 211
@Fourdee I watched last cron job execution, RAM usage is quite low, luckily 😅. Also CPUs are not at 100%. Nice 19 is also good there to in case prioritize web access. I think it would be worse, if we'd really source every file into it's own array first, instead of add info to counting array and reset the other variables again. On the other hand...
root@DietPi:~# G_TREESIZE /home/dietpi-survey/survey/
[ OK ] Root access verified.
5.5 MB /home/dietpi-survey/survey/Can the RAM usage be higher than what all the files actually contain (including comments)?
With v6.10 (and every user doing at least one empty upload), we should add a total count, might be confusing for others as well 😆. 1:5 opted-in vs opted-out would be horrible. Luckily more users decide to stay (although we do not know currently, how much chose option 1: Opt out without sending empty purge file).
MichaIng
commented
6 years ago @Fourdee Tested your 4D version:
root@DietPi:~# time /etc/cron.minutely/survey-report_4D_testing
real 0m38.168s
user 0m3.492s
sys 0m4.092s
root@DietPi:~# time /etc/cron.minutely/survey-report
real 0m44.668s
user 0m3.908s
sys 0m5.372s
root@DietPi:~# nano /etc/cron.minutely/survey-report
root@DietPi:~# time /etc/cron.minutely/survey-report
rm: cannot remove '/tmp/dietpi-survey': No such file or directory
real 0m50.749s
user 0m4.560s
sys 0m6.132sAssociative arrays (declare -A aTEST, supports string indexes):
root@DietPi:~# declare -A aTEST
root@DietPi:~# aTEST[test ing ahhh]=test1
root@DietPi:~# aTEST[hmm mal sehn]=test2
root@DietPi:~# test='jo schaun wir mal'
root@DietPi:~# aTEST[$test]=test3
root@DietPi:~# echo ${!aTEST[@]}
test ing ahhh hmm mal sehn jo schaun wir mal
root@DietPi:~# echo "${!aTEST[@]}"
test ing ahhh hmm mal sehn jo schaun wir mal
root@DietPi:~# echo "${aTEST[@]}"
test1 test2 test3
root@DietPi:~# for i in ${!aTEST[@]}; do echo $i; done
test
ing
ahhh
hmm
mal
sehn
jo
schaun
wir
mal
root@DietPi:~# for i in ${!aTEST[*]}; do echo $i; done
test
ing
ahhh
hmm
mal
sehn
jo
schaun
wir
mal
root@DietPi:~# for i in ${!aTEST[*]}; do echo "$i"; done
test
ing
ahhh
hmm
mal
sehn
jo
schaun
wir
mal
root@DietPi:~# for i in "${!aTEST[*]}"; do echo "$i"; done
test ing ahhh hmm mal sehn jo schaun wir mal
root@DietPi:~# for i in "${!aTEST[@]}"; do echo "$i"; done
test ing ahhh
hmm mal sehn
jo schaun wir mal
root@DietPi:~# for i in "${!aTEST[@]}"; do echo $i; done
test ing ahhh
hmm mal sehn
jo schaun wir mal@MichaIng
Love the dark theme and new fonts, nice 👍
MichaIng
commented
6 years ago @Fourdee As we anyway change the survey, we can think of further or not needed entries:
custom/own/unofficial branch/fork. I don't think it will be much, but could be still interesting, if existent.#!/bin/bash and in other cases, as said we can also just handle based on DietPi version.Fourdee
commented
6 years ago @MichaIng
Interesting could be as well the one or the other chosen setting/dietpi.txt entry: How much users use first run automation, use local or public time servers (after we implemented it) and such things.
This I like!
Automated install data is a must 👍
Fourdee
commented
6 years ago @MichaIng
Is this right, I have no idea how you are going to convert this into the HTML side (I may lack some knowledge here hehe)?
root@DietPi:/home/dietpi-survey/survey# cat 5532a4ec-1001-4ca2-80da-a670217f28ce.txt
#!/bin/bash
# -------------------------
((aSURVEY_VERSION[6]++))
# -------------------------
((aSURVEY_SENTCOUNT[16]++))
((aDIETPI_VERSION[6.10]++))
((aDEVICE_NAME[NanoPC T4 (aarch64)]++))
((aCPU_ARCH[${aCPU_NAME[3]}]++))
((aCPU_COUNT[6]++))
((aDISTRO_VERSION[buster]++))
((aAUTOSTART_OPTION[${aAUTOSTART_NAME[0]}]++))
((aAUTO_SETUP_AUTOMATED[0]++))
((aNETWORK_INTERFACE[eth0]++))
# -------------------------
# DietPi-Software Installed
# -------------------------
((aSOFTWARE[${aSOFTWARE_NAME6_10[0]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_10[103]}]++))
((aSOFTWARE[${aSOFTWARE_NAME6_10[105]}]++))This is how I would of done it, unsure if it helps:
. 5532a4ec-1001-4ca2-80da-a670217f28ce.txt $current_web_array_index
((current_web_array_index++))root@DietPi:/home/dietpi-survey/survey# cat 5532a4ec-1001-4ca2-80da-a670217f28ce.txt
#!/bin/bash
# -------------------------
aSURVEY_VERSION[$1]=6
# -------------------------
...
aNETWORK_INTERFACE[$1]='eth0'
# -------------------------
# DietPi-Software Installed
# -------------------------
aSOFTWARE[$1,0]='Xserver' # Scraped from dietpi-software survey_list_installed_name (to only provide name of installed software into a 2d array format)
aSOFTWARE[$1,1]='Xserver'
aSOFTWARE[$1,2]='Kodi'@Fourdee
Will do testing now, handling is already partly added to survey-report script. As you check the current survey script, the loops already contain those array number increases of type: ((aSURVEY_VERSION[6]++))
When sourcing the file, this is not done automatically to the right array. Values look correct. Translation of arrays to html file does not change, as the arrays are the same that are already created for v6.9.
🈯️ Looks fine. Was this you for testing? DietPi v7.9
🈯️ Next test works as expected. My "TestVirtual Machine..." and v6.11 VMs appear as they should 😃.
MichaIng
commented
6 years ago @Fourdee So far it works quite well. Only thing is that the list is quite long already, much to scroll. Great would be to have separate pages for the different counts/tables, perhaps some overview page/frame with links. The ongoing upload count table would be also not as problematic any more. Currently I would vote to show it somehow different instead of an own row for every single count.
Perfect would be to have it with similar style than https://dietpi.com/dietpi-software.html, integrated into dietpi.com main page.
userdeveloper98
commented
6 years ago Hey guys, I have something for you !
Check it out: https://files.fm/u/skzgfequ
Extract, run (java -jar dietPiSurveyDemo.jar), open http://localhost:8080/survey.xhtml.
(You need java installed and port 8080 free)
Hope you will like it :+1:
Fourdee
commented
6 years ago @userdeveloper98
I tried it using remote connection, hangs on loading white page.
Works great on localhost, looks good 👍 :
How are you scraping the data? Directly from https://dietpi.com/dietpi-software.html?
userdeveloper98
commented
6 years ago @Fourdee
How are you scraping the data?.
I used hardcoded values just for the sake of demo... If you like it we can do a real one...
Fourdee
commented
6 years ago @userdeveloper98
I used hardcoded values just for the sake of demo... If you like it we can do a real one...
By all means, if this is something you want to do and works well, we can add it to the server. Ideally, if you can scrape from https://dietpi.com/dietpi-software.html directly, it would be best, although, if you do go ahead, we could export the data values as a .txt download?
The only issue is I have 0 knowledge of Java, so all maintenance of the page would be down to you.
MichaIng
commented
6 years ago @Fourdee You mean scraping from dietpi.com/survey right? ;)
@userdeveloper98 Looks really nice, great work. Remote access from VM host to VM guest worked all here. Ideally of course would be it running on dietpi.com and directly reading data from survey upload directory to replace current static dietpi.com/survey
Only concerns: It runs as java, so the server needs to have java installed and run it. Took 500M RAM here on my VM. This is a quarter of whole dietpi.com RAM and perhaps too much, just for a meta page. Or is there a way to have it somehow run in a more lightweight state? Would be of course preferred to have it using the available apache webserver, but I guess it is not possible the way it is currently.
And jep, also no deeper java knowledge here as well, but we can help to provide example data, in the way we have it on the server.
userdeveloper98
commented
6 years ago @MichaIng @Fourdee
Ideally, of course, would be it running on dietpi.com and directly reading data from survey upload directory to replace current static dietpi.com/survey
Agree, where is survey upload directory?
Took 500M RAM here on my VM. This is a quarter of whole dietpi.com RAM and perhaps too much, just for a meta page. Or is there a way to have it somehow run in a more lightweight state?
It's a very dirty draft, it contains a lot of things that are not needed. Also, by default, it takes as much as it can for the cache, but it can be fine-tuned to specific requirements A more lightweight version will be built once we agree on all details and preferences 😉
And jep, also no deeper java knowledge here as well, but we can help to provide example data, in the way we have it on the server.
although, if you do go ahead, we could export the data values as a .txt download?
It would be great! It potentially can update itself very efficient. (ex: only when file changes were detected.)
I would prefer to parse a .txt file or .csv or even better a DB query. Scraping HTML is very error prone and minor changes in HTML will require a lot of maintenance work.
P.S (I didn't investigate a lot how the survey works now) Just a thought.. what do you think if we will assign a unique id for each user to maintain survey reports accurate (avoid duplicates, track history, etc...). This should be something that will give confidence that it is really unique and there is nothing related to personal data. A perfect candidate for this is the MAC address in my opinion. It is related to the hardware so it will not change after reinstalls or user configurations. And in addition, just to avoid leaking private data, we will hash MAC address with a strong algorithm. In this way, we will have a unique, depersonalized id for each user that will not change after reinstall or configurations.
MichaIng
commented
6 years ago @userdeveloper98
Agree, where is survey upload directory?
/home/dietpi-survey/survey/something that will give confidence that it is really unique and there is nothing related to personal data
/DietPi/dietpi/.hw_model to get yours. If you check survey creation, this ID is used as upload file name and by this you can uniquely identify also the data within upload folder.With v6.11 we optimized the survey file scheme for our current report creation, thus it is a bid cryptic, containing bash array creation/upcounting code.
When starting dietpi-survey your personal survey file will be shown. Most array indexes are in combination with array name self explained, others are embedded into another array that translates the index into string (autostart index, software index). These might be a bid tricky, as especially software indexes are added and removed on DietPi updates, also autostart indexes might change, however not very often. So you need a version specific translation, which is currently done by the inner array within the survey file.
I will upload the current survey creation file. @Fourdee Is there a way to upload (meta) files to github that must not be downloaded via git archive download or clone, perhaps all files within .github or are just all files with leading dot ignored?
Hmm, actually easy to test with dedicated branch 😄.
Fourdee
commented
6 years ago @MichaIng
@Fourdee Is there a way to upload (meta) files to github that must not be downloaded via git archive download or clone, perhaps all files within .github or are just all files with leading dot ignored? Hmm, actually easy to test with dedicated branch
No idea, buts its a good idea 👍
userdeveloper98
commented
6 years ago @MichaIng
Agree, where is survey upload directory?
/home/dietpi-survey/survey/
I mean the folder where all concatenated survey reports are, from all users. I saw a sample when I opted in, but I don't really know how all reports together will look like. Is that somehow parsed/formatted matched to UniqID or just appended, etc...
I need a sample of the file which application will need to read -> parse -> draw charts. Ideally, it will be the latest version of the file with all users info.
maybe it is best to translate this ID internally to a simply index to show on the page, just that no one can see an ID that is present on your own DietPi
Will be done 👍 .
better a definitely not changeable core hardware ID would be better
I see this as a better option because after re-install you don't really now, is this a new user or an upgraded one? Few re-installs and you will get unrealistic statistics.
MichaIng
commented
6 years ago @userdeveloper98
This gives you the file name of your upload, based on DietPi created hardware ID: echo $(sed -n 5p /DietPi/dietpi/.hw_model).txt
Those uploaded files are placed within the mentioned folder and not changed there. They are just parsed/sourced by the survey-report script to create the static html page.
I just searched a bid about retrieving a bullet prove unique hardware ID, and there simply doesn't seem to be a solution:
dmidecode seems to give hardware IDs as well, some they are sometimes by manufacturer just created of MAC address + leading or trailing zeros and/or finally by the scheme cannot be totally unique (duplications statistically). And dmidecode is not available/working on all systems as well.If we found a totally unique ID that is attached to the motherboard and/or CPU and cannot be manipulated software wise, we could hash this again and then use it for uploads. But as it doesn't seem so trivial, and as well would lead to another dietpi-survey reset (which we just had), I would for now stay with the generated ID. The data is/will be incorrect for completely fresh DietPi installs, but I think this is something we can live with. The for us important ratio between used devices and software titles will be precise enough the check whether development into the one or the other direction will be beneficial for more users.
My idea anyway is, do e.g. drop/handle differently files that were not updated since a certain time, 3 months e.g.. Most likely devices are inactive (or reflashed), if dietpi-update and dietpi-software was never called within 3 months. To use file ages even more precise, we could call dietpi-survey as well within e.g. monthly cron job and then use 1 month as a drop age.
Okay, all files within the repo are downloaded with the zip, there are no meta files possible, AFAIK. Don't know about if there are some other possibilities via releases or other options that can be enabled for the repo to files, that should not be included with the zip.
@Fourdee
Do you otherwise agree to just use .meta or any other folder in repo root to place files such as survey-report or https://github.com/Fourdee/DietPi/blob/dietpi-cloud-migration/dietpi-cloud-migration? A separate branch or even repo is totally overkill. But don't want to mess up the repo either.
Fourdee
commented
6 years ago Do you otherwise agree to just use .meta or any other folder in repo root to place files such as survey-report or https://github.com/Fourdee/DietPi/blob/dietpi-cloud-migration/dietpi-cloud-migration? A separate branch or even repo is totally overkill. But don't want to mess up the repo either.
Yep, by all means 👍 As long as the folder is not installed on DietPi systems, or, can be removed during the ramdisk update.
MichaIng
commented
6 years ago @userdeveloper98 Here the current script we use: https://github.com/Fourdee/DietPi/blob/testing/.meta/survey_report
userdeveloper98
commented
6 years ago @MichaIng Thanks a lot for the script! Now it's crystal clear how everything works. 😃
MichaIng
commented
6 years ago @Fourdee
What do you think about weekly or monthly dietpi-survey 1 executions to allow us removed obsolete uploads?
Fourdee
commented
6 years ago @MichaIng
What do you think about weekly or monthly dietpi-survey 1 executions to allow us removed obsolete uploads?
Weekly cron job to automate sending of the survey if opted in?
Not sure, I doubt users would be welcomed to this. Ideally, it should only be run during installs/updates?
MichaIng
commented
6 years ago @Fourdee Jep, also sending empty file if opt out. I don't see another way to safely identify obsolete uploads.
Fourdee
commented
6 years ago @MichaIng
Jep, also sending empty file if opt out. I don't see another way to safely identify obsolete uploads.
Yep, same here. We lack any method of unlinking the device, when its no longer in use. The only thing I can think of is a simple "This system is no longer going to be used" option, which would remove the survey, however, relies on the user doing it, and, not abusing it.
MichaIng
commented
6 years ago @Fourdee Nothing we can rely on. I mean it's not a really bid issue, the ratios between used devices and software will be not affected too much, and we could simply assume a max time before marking the related upload as obsolete, e.g. 3 months (?), so old files will not grow unlimited. But for statistic and development reasons and the possibilities with @userdeveloper98 's solution, we should not remove, but move them into an "old" folder for now. Still some time until survey v5 reaches 3 months anyway 😉.
userdeveloper98
commented
6 years ago @MichaIng @Fourdee I need your feedback on this: https://files.fm/u/wpvsdrc5 This is another draft that comes with a set of survey samples. You can repoint via properties file to the real folder and check if it is parsing properly the files. Parsing is generic, you can adjust the template to parse any text file. So that changing in future file structure will not require code changes. Let me know if you want specific charts/filters/searches etc.. Once we finish with desired features I will work on performance optimisations.
Fourdee
commented
6 years ago Notes:
Open: [http://localhost:8888/survey.xhtml]. And enjoy :)@userdeveloper98
Good progress 👍 Scraping seems to work well.
Not sure if you can use custom fonts with Java? But if you can, the font DietPi logo is based on is: https://cooltext.com/Download-Font-Michroma
Might work well for the UI?
In regards to the "Devices and installed versions barchart". Personally, i'd simplify this and have only "Install count per device"?
In the survey database tab, some suggestions:
userdeveloper98
commented
6 years ago @Fourdee Thanks for the feedback. New version released: [link]
Fourdee
commented
6 years ago @userdeveloper98
Nice progress 👍
Some issues/feedback if needed:
.SurveyLoaderImpl : Mathing software records from survey files
2018-07-28 22:09:55.057 ERROR 12164 --- [Pool-1-worker-4] c.o.d.s.i.file.dao.DpiFileReaderImpl : Ups ! Something wrong on reading file [./data/surveyFiles/PaxHeader]. Cause:[java.io.IOException: Is a directory]
2018-07-28 22:10:11.846 ERROR 12164 --- [ main] c.o.d.s.i.file.loaders.SurveyLoaderImpl : Exception on paralel processing of files. Cause: {}java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006) ~[na:1.8.0_171]
at com.osc98.dpisurvey.service.implementation.file.loaders.SurveyLoaderImpl.insertDoubleParalell(SurveyLoaderImpl.java:83) [classes!/:0.0.2-SNAPSHOT]
at com.osc98.dpisurvey.service.implementation.file.loaders.SurveyLoaderImpl.insertRecords(SurveyLoaderImpl.java:51) [classes!/:0.0.2-SNAPSHOT]
at com.osc98.dpisurvey.service.implementation.file.loaders.SurveyLoaderImpl$$FastClassBySpringCGLIB$$739cd35e.invoke(
Hi @Fourdee
On the main barchart for "devices and installed version". I strongly feel a simple "Device install count" (without versions) would make this cleaner for the end user to read (less busy)?
Something failed when running the java program
(Seems like inside surveyFiles folder where another folder, while it was expected only files :disappointed: on reading file [./data/surveyFiles/PaxHeader]. Cause:[java.io.IOException: Is a directory] )
CPU arch selection has incorrect results for me
Can't find what it can be :cry: Is this happening on provided samples or on real data? (samples are generated absolutely random, so you may see for example that RasspberyPi have 8 cores while in reality, this model have 4 cores, the same go for cpu arch)
userdeveloper98
commented
6 years ago v 1.0.0 Should be ready to be deployed as it is, next iteration will be minor performance optimizations. New features:
Fourdee
commented
6 years ago @userdeveloper98
Nice one, good progress 👍
Some feedback from my end:
, and the additional number mean in this. Is it a total of 3?
userdeveloper98
commented
6 years ago @MichaIng v1.0.1
3,21 it will show Count: 3#D2D2D2. Looks nicer.#C2DE60)Fourdee
commented
6 years ago @userdeveloper98
Awesome, great work, looking the part now 👍
I've created a local system for testing, contains all the survey files: http://dietpi-4d.ddns.net:8888/survey EDIT: its restarting, should be up any minute.
Some issues noted:
2018-08-04 07:12:37.821 ERROR 21138 --- [onPool-worker-2] c.o.d.s.i.file.dao.SurveyFileReaderImpl : UPS ! templateLine:[((aSOFTWARE[${aSOFTWARE_NAME6_11[<softwareIndex>]}]++))]. surveyLine[((aSOFTWARE[${aSOFTWARE_NAME6_11[]}]++))]. Cause:[java.lang.NumberFormatException: For input string: ""]
2018-08-04 07:12:38.032 ERROR 21138 --- [onPool-worker-2] c.o.d.s.i.file.dao.SurveyFileReaderImpl : UPS ! templateLine:[((aSOFTWARE[${aSOFTWARE_NAME6_11[<softwareIndex>]}]++))]. surveyLine[((aSOFTWARE[${aSOFTWARE_NAME6_11[]}]++))]. Cause:[java.lang.NumberFormatException: For input string: ""]
Aside from the above, this is really coming along now, very impressive 👍. If we can get this finished up, i'll put this on the main server. Does the program automatically re-scan/update the survey database during runtime? If not, I can create a service with cron job to auto restart every X hours etc.
As we restart the new updated safer and more transparent DietPi-Survey (https://github.com/Fourdee/DietPi/pull/1822), I though it would be nice to have a public result page to review current statistics. So users know exactly what we do with the data and we can also easily access them without the need to do manual extraction, sorting and evaluation.
I though about some cron job, that extracts the data, sorts them and writes them back to some simple HTML file that can be accessed publicly.
Extremely nice would be of course a ready to use PHP table script with sorting, filtering and search function, but I couldn't find one through quick www research. The latter one would e.g. need the data to be regularly read and saved into a MariaDB table from where the PHP script can read. If someone knows how to achieve this or has fun to write such script, would be welcome.
The installed software part could be a bid tricky, as the software IDs change by times. I thought about another (array) file for each DietPi version that connects software IDs with software title.
This is how the survey files currently look like:
Ah @Fourdee we should remove hostname for anonymity? Can't think of value we get through this.