Closed whedon closed 3 years ago
amanchokshi
commented
3 years ago I've created a new Performance page in the documentation. This outlines our investigation into parallel and single core performance and states the I/O limitations. I've also added a link to the performance page in the examples.
teuben
commented
3 years ago I've removed all directories (except TLE) in embers_out/sat_utils, the TLE directory has the following 73 files:
21576.txt 25113.txt 25117.txt 25159.txt 25415.txt 25419.txt 25477.txt 25481.txt 25982.txt 25986.txt 33062.txt 40069.txt 40089.txt 41180.txt 41184.txt 41188.txt 44020.txt 44025.txt 44387.txt 23545.txt 25114.txt 25118.txt 25338.txt 25416.txt 25420.txt 25478.txt 25482.txt 25983.txt 28654.txt 33063.txt 40086.txt 40090.txt 41181.txt 41185.txt 41189.txt 44022.txt 44026.txt 23546.txt 25115.txt 25119.txt 25413.txt 25417.txt 25475.txt 25479.txt 25980.txt 25984.txt 33060.txt 33064.txt 40087.txt 40091.txt 41182.txt 41186.txt 44018.txt 44023.txt 44027.txt 25112.txt 25116.txt 25158.txt 25414.txt 25418.txt 25476.txt 25480.txt 25981.txt 25985.txt 33061.txt 33065.txt 40088.txt 41179.txt 41183.txt 41187.txt 44019.txt 44024.txt 44028.txt
I then ran ephem_batch --max_cores=1 and only noticed some warnings
sat_ephemeris.py:286: RuntimeWarning: More than 20 figures have been opened.
Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed and may consume too much memory.
Running ephem_chrono immediately returns (though it has the coffee message)
$ ephem_chrono Saving chronological Ephem files to: ./embers_out/sat_utils/ephem_chrono Grab a coffee, this may take more than a couple of minutes!
It does create all the json files, but they just contain [], and the log file has 0 lines.
I am doing this in a clean directory, where i only see embers_out and tiles_data. The code is somewhere else, which I installed using "pip install -e ." I am using a "venv", which is a directory inside that EMBERS source tree.
amanchokshi
commented
3 years ago This is bizarre. Let try a couple of long shots. Can you check whether the files in the TLE directory contain anything? Perhaps they've been corrupted somehow?
You should see something like this:
$ tail embers_out/sat_utils/TLE/21576.txt
1 21576U 91050C 19282.72964883 .00000014 00000-0 18223-4 0 9990
2 21576 98.6203 331.5014 0002898 326.0593 34.0410 14.41572163482935
1 21576U 91050C 19283.90959667 +.00000017 +00000-0 +19271-4 0 9992
2 21576 098.6201 332.6919 0002839 321.6987 038.4000 14.41572293482883Have you been running all the other examples in the same way? With EMBERS installed using pip install -e . in the source tree?
I've attempted to replicate the way you're running things. I have my source directory at ~/Repos/EMBERS within which I've created a virtual environment and run pip install --editable . If I do a pip list when this virtual environment is active, I see that embers version 0.8.2 is installed form source
Package Version Location
----------------------------- ----------- -----------------------------------
embers 0.8.2 /Users/amanchokshi/Repos/EMBERS/srcI've then created a new directory ~/Repos/em_test/ which just contains tiles_data and embers_out/sat_utils/TLE
Keeping the original virtual env active, I run the following
$ ephem_batch
$ ephem_chronoChecking the contents of the first file in the ephem_chrono dir:
$ tail embers_out/sat_utils/ephem_chrono/2019-10-10-00:00.json
1.5377585726597032,
1.5366094819324243,
1.5354642595022225,
1.534322886123595,
1.5331853425510376,
1.5320516095390462,
1.5309216678421183
]
}
]% Have I deviated from your setup in any way?
PS: It does usually take a while to process the data (~10 mins), if we were working with a larger data set. The sample data only contains a couple of satellite passes each. I'll remove the the coffee message in the next EMBERS version.
amanchokshi
commented
3 years ago So many matplotlib figures should not be opened by ephem_batch, instead figures should be saved to embers_out/sat_utils/ephem_plots. Does ephem_batch create this directory and .png figures for you?
Can you update EMBERS to V0.8.3, or git pull and build from source again? I've explicitly closed the figures after the plot is saved. This shouldn't be necessary, but may help.
Also have removed the coffee message, which was unnecessary
teuben
commented
3 years ago Indeed my embers_out/sat_utils/TLE/21576.txt looks like yours. I've been able to run all steps before this fatal one just fine, all the figure are like yours.
You have also done what I did , with the pip installing it with source in place. I use a pretty standard anaconda3 as the base for my python, not the system python. I've actually reinstalled that one.
I will retrace the steps, but a question before I do this, can i reinstall and just do ephem_batch, it doesn't need the lengthy align_batch, does it ?
A suggestion on your side might be to have some kind of -v flag for all your scripts, which makes it verbose, to perhaps help to see some expected intermediate output, helping you to see what's going on here. In my own system I have debug=LEVEL, which will create more and more output the higher the level is. Other packages have -vvv instead of -v to increase the verbosity level :[) Even for myself I find it useful, as I forget my own code, and a debug level could help with that.
teuben
commented
3 years ago So many
matplotlibfigures should not be opened byephem_batch, instead figures should be saved toembers_out/sat_utils/ephem_plots. Doesephem_batchcreate this directory and.pngfigures for you?Can you update EMBERS to V0.8.3, or
git pulland build from source again? I've explicitly closed the figures after the plot is saved. This shouldn't be necessary, but may help.Also have removed the coffee message, which was unnecessary
I noted that 3 files in TLE have 3 length:44019.txt 44020.txt 44025.txt - the rest have anywhere from 20 to 78 lines. There are 73 files in here.
I do see 70 plots in the ephem_plots directory. (not 73, since 3 of my TLE's were 0 length)
I've updated code again. Ran ephem_chrone, but still no json files.
amanchokshi
commented
3 years ago You don't need to run align_batch again. The only thing that any of the ephem scripts should need are the TLE files. It's also okay that 3 of the TLE files are empty, they have very sparse satellite coverage and on that data had np passes visible from the site.
I've been using pyenv to install python versions on my system. Currently I've used pyenv to set by default version to 3.8.3. I used pyenv virtualenv to create my virtual enviroments. I don't think this should be an issue.
Are you on a python version >= 3.6? I have used f-strings, do 3.5 will have problems
amanchokshi
commented
3 years ago Just a thought, is the directory with tiles_data and embers_out/sat_utils/TLE on your external VFAT formatted drive? The names of the json files created have a colon : in them. Could that be tripping things up?
teuben
commented
3 years ago You don't need to run
align_batchagain. The only thing that any of theephemscripts should need are the TLE files. It's also okay that 3 of the TLE files are empty, they have very sparse satellite coverage and on that data had np passes visible from the site.I've been using
pyenvto install python versions on my system. Currently I've usedpyenvto set by default version to3.8.3. I usedpyenv virtualenvto create my virtual enviroments. I don't think this should be an issue.Are you on a python version >= 3.6? I have used f-strings, do 3.5 will have problems
python 3.8.5 the whole tree with embers_out and tiles_data is on a regular ext4 formatted drive. The json files look fine in their name, e.g. -rw-rw-r-- 1 teuben teuben 2 Nov 19 09:00 embers_out/sat_utils/ephem_chrono/2019-10-10-00:00.json
amanchokshi
commented
3 years ago Lets rule out problems in ephem_batch so that we can focus on ephem_chrono. Can you try and see if the log seems similar?
$ tail embers_out/sat_utils/ephem_data/ephem_batch.log
INFO: ephem_batch: Saved sky coverage plot of satellite [25481] to ./embers_out/sat_utilsephem_plots/25481.png
Saved ephemeris of satellite [25481] to ./embers_out/sat_utilsephem_data/25481.npz
INFO: ephem_batch: Saved sky coverage plot of satellite [23546] to ./embers_out/sat_utilsephem_plots/23546.png
Saved ephemeris of satellite [23546] to ./embers_out/sat_utilsephem_data/23546.npz
INFO: ephem_batch: Saved sky coverage plot of satellite [25479] to ./embers_out/sat_utilsephem_plots/25479.png
Saved ephemeris of satellite [25479] to ./embers_out/sat_utilsephem_data/25479.npz
INFO: ephem_batch: Saved sky coverage plot of satellite [23545] to ./embers_out/sat_utilsephem_plots/23545.png
Saved ephemeris of satellite [23545] to ./embers_out/sat_utilsephem_data/23545.npz
INFO: ephem_batch: Saved sky coverage plot of satellite [25478] to ./embers_out/sat_utilsephem_plots/25478.png
Saved ephemeris of satellite [25478] to ./embers_out/sat_utilsephem_data/25478.npzAlso that the first plot embers_out/sat_utils/ephem_plots/21576.png looks like this:
And finally that the size of the ephem_data directory matches mine:
$ du -sh embers_out/sat_utils/ephem_data
>>> 2.2M embers_out/sat_utils/ephem_datalog file looks ok, but has a different order of the files downloaded. File has 143 lines (73 for files, 70 for successfull ones)
the plot looks similar, though mine has 41 passes, yours has 5.
my du -sh embers_out/sat_utils/ephem_data 17M embers_out/sat_utils/ephem_data
e.g. the one you plotted, where I had 41 passes:
-rw-rw-r-- 1 teuben teuben 146855 Nov 19 08:26 21576.npz
It has 70 files (3 missing due to 0 length TLE files)
On 11/19/20 9:36 AM, Aman Chokshi wrote:
Lets rule out problems in |ephem_batch| so that we can focus on |ephem_chrono|. Can you try and see if the log seems similar?
|$ tail embers_out/sat_utils/ephem_data/ephem_batch.log INFO: ephem_batch: Saved sky coverage plot of satellite [25481] to ./embers_out/sat_utilsephem_plots/25481.png Saved ephemeris of satellite [25481] to ./embers_out/sat_utilsephem_data/25481.npz INFO: ephem_batch: Saved sky coverage plot of satellite [23546] to ./embers_out/sat_utilsephem_plots/23546.png Saved ephemeris of satellite [23546] to ./embers_out/sat_utilsephem_data/23546.npz INFO: ephem_batch: Saved sky coverage plot of satellite [25479] to ./embers_out/sat_utilsephem_plots/25479.png Saved ephemeris of satellite [25479] to ./embers_out/sat_utilsephem_data/25479.npz INFO: ephem_batch: Saved sky coverage plot of satellite [23545] to ./embers_out/sat_utilsephem_plots/23545.png Saved ephemeris of satellite [23545] to ./embers_out/sat_utilsephem_data/23545.npz INFO: ephem_batch: Saved sky coverage plot of satellite [25478] to ./embers_out/sat_utilsephem_plots/25478.png Saved ephemeris of satellite [25478] to ./embers_out/sat_utilsephem_data/25478.npz |
Also that the first plot |embers_out/sat_utils/ephem_plots/21576.png| looks like this:
21576 https://user-images.githubusercontent.com/40157844/99679853-881b8080-2ad0-11eb-889c-1b4341201360.png
And finally that the size of the |ephem_data| directory matches mine:
|$ du -sh embers_out/sat_utils/ephem_data >>> 2.2M embers_out/sat_utils/ephem_data |
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2629#issuecomment-730415928, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ4MGNROWLMCID7EYTE4S3SQUUNXANCNFSM4QSAJ3TA.
amanchokshi
commented
3 years ago It's really strange that you have 41 satellite passes. Can you try this
$ ephem_single --sat=21576 --out_dir=tmp
$ tree tmp
tmp
├── ephem_data
│ └── 21576.npz
└── ephem_plots
└── 21576.pngThe .npz data file should be only 21K in size. If the plot has more than 5 satellites, it means that something is wrong with the TLE files. This would explain why ephem_chrono is making empty files
amanchokshi
commented
3 years ago It's a long shot, but can you check if the contents of the TLE file is the same as mine:
$ more embers_out/sat_utils/TLE/21576.txt
1 21576U 91050C 19282.72964883 .00000014 00000-0 18223-4 0 9990
2 21576 98.6203 331.5014 0002898 326.0593 34.0410 14.41572163482935
1 21576U 91050C 19283.90959667 +.00000017 +00000-0 +19271-4 0 9992
2 21576 098.6201 332.6919 0002839 321.6987 038.4000 14.41572293482883
ls -l tmp/ephem_/ -rw-rw-r-- 1 teuben teuben 146855 Nov 19 10:18 tmp/ephem_data/21576.npz -rw-rw-r-- 1 teuben teuben 171585 Nov 19 10:18 tmp/ephem_plots/21576.png
so much bigger. The png shows 41 passes.
On 11/19/20 10:07 AM, Aman Chokshi wrote:
It's really strange that you have 41 satellite passes. Can you try this
|$ ephem_single --sat=21576 --out_dir=tmp $ tree tmp tmp ├── ephem_data │ └── 21576.npz └── ephem_plots └── 21576.png |
The |.npz| data file should be only |21K| in size. If the plot has more than 5 satellites, it means that something is wrong with the TLE files. This would explain why |ephem_chrono| is making empty files
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2629#issuecomment-730436600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ4MGLXEJYJN6DEX5NKB7LSQUYDRANCNFSM4QSAJ3TA.
teuben
commented
3 years ago mine is clearly longer
more embers_out/sat_utils/TLE/21576.txt 1 21576U 91050C 19274.12296826 .00000022 00000-0 20647-4 0 9992 2 21576 98.6215 322.8165 0003370 355.9403 4.1757 14.41571172481698 1 21576U 91050C 19274.74764681 +.00000022 +00000-0 +20705-4 0 9994 2 21576 098.6214 323.4469 0003330 354.0023 006.1125 14.41571283481565 1 21576U 91050C 19275.64996028 .00000018 00000-0 19615-4 0 9999 2 21576 98.6213 324.3575 0003275 351.1025 9.0105 14.41571404481914 1 21576U 91050C 19276.69109109 .00000011 00000-0 17383-4 0 9994 2 21576 98.6211 325.4081 0003216 347.5805 12.5303 14.41571520482061 1 21576U 91050C 19277.73222188 .00000003 00000-0 14745-4 0 9993 2 21576 98.6209 326.4587 0003154 343.9674 16.1414 14.41571600482214 1 21576U 91050C 19278.70394390 -.00000000 +00000-0 +13856-4 0 9996 2 21576 098.6208 327.4393 0003103 340.6734 019.4337 14.41571694482138 1 21576U 91050C 19279.12039615 .00000001 00000-0 14345-4 0 9993 2 21576 98.6207 327.8595 0003082 339.1436 20.9627 14.41571750482416 1 21576U 91050C 19279.74507450 +.00000006 +00000-0 +15778-4 0 9994 2 21576 098.6207 328.4898 0003052 336.8952 023.2099 14.41571851482289 1 21576U 91050C 19280.36975290 .00000009 00000-0 16766-4 0 9996 2 21576 98.6206 329.1202 0003016 334.7424 25.3617 14.41571918482593 1 21576U 91050C 19280.71679643 +.00000012 +00000-0 +17538-4 0 9990 2 21576 098.6206 329.4703 0002999 333.5040 026.5995 14.41571968482421 1 21576U 91050C 19281.68851827 .00000014 00000-0 18394-4 0 9996 2 21576 98.6205 330.4508 0002950 329.8875 30.2144 14.41572084482783 1 21576U 91050C 19282.72964883 .00000014 00000-0 18223-4 0 9990 2 21576 98.6203 331.5014 0002898 326.0593 34.0410 14.41572163482935
On 11/19/20 10:11 AM, Aman Chokshi wrote:
It's a long shot, but can you check if the contents of the TLE file is the same as mine:
|$ more embers_out/sat_utils/TLE/21576.txt 1 21576U 91050C 19282.72964883 .00000014 00000-0 18223-4 0 9990 2 21576 98.6203 331.5014 0002898 326.0593 34.0410 14.41572163482935 1 21576U 91050C 19283.90959667 +.00000017 +00000-0 +19271-4 0 9992 2 21576 098.6201 332.6919 0002839 321.6987 038.4000 14.41572293482883 |
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2629#issuecomment-730439126, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ4MGL4ZM7JLITBB3OK6FLSQUYRPANCNFSM4QSAJ3TA.
amanchokshi
commented
3 years ago The data you have doesn't match what's on github currently. I initially had too much data on the sample data repo and had to reduce it to make the examples run in a reasonable time.
Can you pull down the sample data repo again, or just do download the TLE dir from the sample data repo?
The new TLE files should be sufficient to get ephem_chrono to work.
Can you check whether all the files in tiles_data have the time-stamp 2019-10-10 in them? If they do, you have the correct data and the wrong TLE files. If they don't you'll unfortunately have do pull the whole sample data repo again.
Sorry for the hassle. Hopefully it's smooth sailing from here 🤞
teuben
commented
3 years ago I redid the data repo instructions. I now have a short embers_out/sat_utils/TLE/21576.txt file, with 4 lines. The tiles_data are all (1440 I see) from 2019-10-10
I'm afraid, even with a new align_batch and ephem_batch I still get a blank ephem_chrono.
I think I'm going to redo the whole workflow.
amanchokshi
commented
3 years ago Maybe you could make a bash script with just the important stuff and let it run?
#!/bin/bash
align_batch --max_cores=1
ephem_batch --max_cores=1
ephem_chronoAlthough I had done exactly those steps, now it's producing non-zero json files. I see 48 json files, of which the first 16 have a good size. Until 2019-10-10-07:30 The ones from 2019-10-10-08:00.json to 2019-10-10-23:30.json are blanked.
teuben
commented
3 years ago If we can fix this last item, we're done with the review!
amanchokshi
commented
3 years ago Hi Peter, the last files which are empty are the result of our small sample data set. I'm appending a screenshot of all ephem_chrono json files produced when I use the full data set.
teuben
commented
3 years ago Ah, then this is the good news I was waiting for. Basically the examples work, which was the last checkmark for acceptance.
On Fri, Nov 20, 2020, 18:45 Aman Chokshi notifications@github.com wrote:
Hi Peter, the last files which are empty are the result of our small sample data set. I'm appending a screenshot of all ephem_chrono json files produced when I use the full data set.
[image: Screen Shot 2020-11-21 at 10 44 21 am] https://user-images.githubusercontent.com/40157844/99860212-8643e100-2be6-11eb-831c-dd092918e0b0.png
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2629#issuecomment-731463589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ4MGLV7POLH52XA3XYHCLSQ35RTANCNFSM4QSAJ3TA .
mbobra
commented
3 years ago @amanchokshi We're almost there! Can you please archive your release on Zenodo to obtain a DOI and then put that in your README.rst file? Please make sure the authors listed on the Zenodo entry match those listed on the JOSS paper. After that I think we're done 🎉
amanchokshi
commented
3 years ago @mbobra Thanks Monica! I've uploaded EMBERS to Zenodo and have added the DOI to the README.rst :)
teuben
commented
3 years ago fantastic, congrats all
mbobra
commented
3 years ago @whedon check references
whedon
commented
3 years ago Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1017/pasa.2018.30 is OK
MISSING DOIs
- None
INVALID DOIs
- None@whedon generate pdf
whedon
commented
3 years ago :point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
mbobra
commented
3 years ago @whedon set 10.5281/zenodo.4287813 as archive
whedon
commented
3 years ago OK. 10.5281/zenodo.4287813 is the archive.
mbobra
commented
3 years ago @whedon set v1.0.0 as version
whedon
commented
3 years ago OK. v1.0.0 is the version.
mbobra
commented
3 years ago @openjournals/joss-eics This paper is ready for final processing 🎉 Congratulations @amanchokshi!
Thank you very much for your thorough review, @teuben. Your contributions keep JOSS running. I really appreciate all your hard work ☀️
amanchokshi
commented
3 years ago Thanks a lot @teuben and @mbobra, I've learnt a lot though the review process and EMBERS has really improved from all your feedback!
arfon
commented
3 years ago @whedon accept
whedon
commented
3 years ago Attempting dry run of processing paper acceptance...Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1017/pasa.2018.30 is OK
MISSING DOIs
- None
INVALID DOIs
- None:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.
Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1939
If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1939, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.
@whedon accept deposit=true@whedon accept deposit=true
whedon
commented
3 years ago Doing it live! Attempting automated processing of paper acceptance...🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦
whedon
commented
3 years ago 🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨
Here's what you must now do:
Party like you just published a paper! 🎉🌈🦄💃👻🤘
Any issues? Notify your editorial technical team...
arfon
commented
3 years ago @teuben, @mbobra - many thanks for your reviews here and @mbobra for editing too! JOSS simply wouldn't work without the volunteer efforts of folks like yourselves ✨
@amanchokshi - your paper is now accepted and published in JOSS :zap::rocket::boom:
whedon
commented
3 years ago :tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:
If you would like to include a link to your paper from your README use the following code snippets:
Markdown:
[](https://doi.org/10.21105/joss.02629)
HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02629">
<img src="https://joss.theoj.org/papers/10.21105/joss.02629/status.svg" alt="DOI badge" >
</a>
reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02629/status.svg
:target: https://doi.org/10.21105/joss.02629This is how it will look in your documentation:
We need your help!
Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:
Submitting author: @amanchokshi (Aman Chokshi) Repository: https://github.com/amanchokshi/EMBERS Version: v1.0.0 Editor: @mbobra Reviewers: @teuben, @mbobra Archive: 10.5281/zenodo.4287813
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@teuben & @mbobra, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @mbobra know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @teuben
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @mbobra
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper