search

kiyo-masui / bitshuffle

Filter for improving compression of typed binary data.
Other
215 stars 76 forks source link

making openmp support optional, with a flag in setup.py #39

Closed leosala closed 8 years ago

leosala commented 8 years ago

Hallo,

this pull request is just because OpenMP interferes with how we are using this module atm (within a multi-threaded software). It just introduces a flag for enabling OpenMP or not.

It also takes into account the case reported in the pull request #38 by @graeme-winter . Hope this can be useful, let me know if you prefer this to be implemented differently

Thanks!

kiyo-masui commented 8 years ago

Please switch the default to True as most users will want OMP enabled.

Thanks for the mac fix. Still don't understand why clang doesn't support OMP.

leosala commented 8 years ago

done, sorry that slipped through. The mac fix I just copied from #38 , thought it would have been easier to merge the two requests as are aimed to a similar purpose

simongregorebner commented 8 years ago

I would like to vote for the default FALSE as we today found out that having openmp enabled will cause (strange) problems due to openmp "optimisations" on multicore systems. At the end of this comment you will find a lengthy test report of one of our measurements (this behaviour is all time reproducible).

Our setup is as follows:

The times shown in the log (the number at the very end), show the raw time spend for the decompression (in seconds). As you can see, periodically (but steadily) we get extraordinary high times for the decompression. This time is obviously caused by "optimisations" (locks/blocks) introduced by openmp. (using bitshuffle without openmp all times are ok)

However its on you to decide on the default. I would already be very happy to have this switch/pull-request in.

#### /var/log/spotter_1.err

2016-04-05 13:44:00,767 - DEBUG - post-decoding nbytes 3546827 0.24
2016-04-05 13:44:07,582 - DEBUG - post-decoding nbytes 3561509 0.13
2016-04-05 13:44:22,365 - DEBUG - post-decoding nbytes 3569240 0.13
2016-04-05 13:44:32,000 - DEBUG - post-decoding nbytes 3545009 0.08
2016-04-05 13:44:53,552 - DEBUG - post-decoding nbytes 3550688 0.09
2016-04-05 13:45:19,652 - DEBUG - post-decoding nbytes 3552871 0.05
2016-04-05 13:45:48,923 - DEBUG - post-decoding nbytes 3562010 0.09
2016-04-05 13:46:00,663 - DEBUG - post-decoding nbytes 3546899 0.06
2016-04-05 13:47:05,669 - DEBUG - post-decoding nbytes 3554393 56.04
2016-04-05 13:47:27,505 - DEBUG - post-decoding nbytes 3561798 0.06
2016-04-05 13:47:50,934 - DEBUG - post-decoding nbytes 3561163 0.08
2016-04-05 13:48:09,869 - DEBUG - post-decoding nbytes 3549771 0.05
2016-04-05 13:48:30,040 - DEBUG - post-decoding nbytes 3553045 0.07
2016-04-05 13:48:56,455 - DEBUG - post-decoding nbytes 4804417 0.08
2016-04-05 13:49:56,916 - DEBUG - post-decoding nbytes 4879966 50.53
2016-04-05 13:50:05,941 - DEBUG - post-decoding nbytes 3548136 0.03
2016-04-05 13:50:24,191 - DEBUG - post-decoding nbytes 3545641 0.10
2016-04-05 13:50:52,218 - DEBUG - post-decoding nbytes 3683731 0.12
2016-04-05 13:51:16,397 - DEBUG - post-decoding nbytes 4565566 0.10
2016-04-05 13:51:29,783 - DEBUG - post-decoding nbytes 3941836 0.12
2016-04-05 13:51:46,154 - DEBUG - post-decoding nbytes 3562649 0.08
2016-04-05 13:52:36,344 - DEBUG - post-decoding nbytes 3681332 29.75
2016-04-05 13:52:55,272 - DEBUG - post-decoding nbytes 4439447 0.10

#### /var/log/spotter_2.err

2016-04-05 13:44:01,369 - DEBUG - post-decoding nbytes 3549379 0.62
2016-04-05 13:44:07,793 - DEBUG - post-decoding nbytes 3539217 0.26
2016-04-05 13:44:22,293 - DEBUG - post-decoding nbytes 3550822 0.08
2016-04-05 13:44:33,277 - DEBUG - post-decoding nbytes 3554660 0.07
2016-04-05 13:44:57,868 - DEBUG - post-decoding nbytes 3545229 0.12
2016-04-05 13:45:23,642 - DEBUG - post-decoding nbytes 3545391 0.08
2016-04-05 13:45:47,597 - DEBUG - post-decoding nbytes 3556207 0.06
2016-04-05 13:46:00,366 - DEBUG - post-decoding nbytes 3555924 0.08
2016-04-05 13:46:31,444 - DEBUG - post-decoding nbytes 3544510 23.67
2016-04-05 13:47:49,668 - DEBUG - post-decoding nbytes 3555234 61.26
2016-04-05 13:48:07,104 - DEBUG - post-decoding nbytes 3557106 0.10
2016-04-05 13:48:26,618 - DEBUG - post-decoding nbytes 3551445 0.10
2016-04-05 13:48:53,595 - DEBUG - post-decoding nbytes 3563193 0.09
2016-04-05 13:50:01,172 - DEBUG - post-decoding nbytes 3548827 45.17
2016-04-05 13:50:50,818 - DEBUG - post-decoding nbytes 4792182 36.89
2016-04-05 13:51:07,174 - DEBUG - post-decoding nbytes 4776790 0.14
2016-04-05 13:51:23,305 - DEBUG - post-decoding nbytes 3563219 0.08
2016-04-05 13:51:44,440 - DEBUG - post-decoding nbytes 3552869 0.08
2016-04-05 13:52:05,190 - DEBUG - post-decoding nbytes 3561976 0.09
2016-04-05 13:52:26,694 - DEBUG - post-decoding nbytes 4980883 0.07
2016-04-05 13:52:37,424 - DEBUG - post-decoding nbytes 4937564 0.06
2016-04-05 13:52:47,824 - DEBUG - post-decoding nbytes 3561337 0.07

#### /var/log/spotter_3.err

2016-04-05 13:44:01,155 - DEBUG - post-decoding nbytes 3555464 0.41
2016-04-05 13:44:07,700 - DEBUG - post-decoding nbytes 3551583 0.07
2016-04-05 13:44:21,317 - DEBUG - post-decoding nbytes 3556226 0.07
2016-04-05 13:45:20,406 - DEBUG - post-decoding nbytes 3553797 44.90
2016-04-05 13:45:44,224 - DEBUG - post-decoding nbytes 3534733 0.08
2016-04-05 13:45:59,127 - DEBUG - post-decoding nbytes 3557105 0.06
2016-04-05 13:46:05,685 - DEBUG - post-decoding nbytes 3571124 0.09
2016-04-05 13:46:21,702 - DEBUG - post-decoding nbytes 3557849 0.10
2016-04-05 13:47:38,041 - DEBUG - post-decoding nbytes 3548885 55.13
2016-04-05 13:49:00,835 - DEBUG - post-decoding nbytes 3552134 61.10
2016-04-05 13:49:26,601 - DEBUG - post-decoding nbytes 3561347 0.12
2016-04-05 13:49:55,962 - DEBUG - post-decoding nbytes 3558971 0.10
2016-04-05 13:50:11,811 - DEBUG - post-decoding nbytes 4014250 0.05
2016-04-05 13:50:27,747 - DEBUG - post-decoding nbytes 3536397 0.05
2016-04-05 13:50:52,541 - DEBUG - post-decoding nbytes 3559642 0.11
2016-04-05 13:51:20,928 - DEBUG - post-decoding nbytes 3551128 0.08
2016-04-05 13:51:41,971 - DEBUG - post-decoding nbytes 3651582 0.06
2016-04-05 13:52:00,048 - DEBUG - post-decoding nbytes 3558799 0.05
2016-04-05 13:52:21,824 - DEBUG - post-decoding nbytes 4719697 0.05
2016-04-05 13:52:33,744 - DEBUG - post-decoding nbytes 4633761 0.09

#### /var/log/spotter_4.err

2016-04-05 13:44:00,479 - DEBUG - post-decoding nbytes 3547491 0.16
2016-04-05 13:44:26,750 - DEBUG - post-decoding nbytes 3558489 18.38
2016-04-05 13:45:24,086 - DEBUG - post-decoding nbytes 3543136 42.61
2016-04-05 13:45:48,032 - DEBUG - post-decoding nbytes 3547740 0.08
2016-04-05 13:46:01,023 - DEBUG - post-decoding nbytes 3560132 0.06
2016-04-05 13:46:10,326 - DEBUG - post-decoding nbytes 3548593 0.05
2016-04-05 13:46:28,767 - DEBUG - post-decoding nbytes 3556804 0.09
2016-04-05 13:46:49,225 - DEBUG - post-decoding nbytes 3556712 0.08
2016-04-05 13:47:14,248 - DEBUG - post-decoding nbytes 3544111 0.04
2016-04-05 13:47:37,111 - DEBUG - post-decoding nbytes 4351664 0.05
2016-04-05 13:47:51,619 - DEBUG - post-decoding nbytes 3554951 0.09
2016-04-05 13:48:09,878 - DEBUG - post-decoding nbytes 3548865 0.06
2016-04-05 13:49:50,004 - DEBUG - post-decoding nbytes 3558575 79.19
2016-04-05 13:50:07,931 - DEBUG - post-decoding nbytes 4804729 0.06
2016-04-05 13:50:15,277 - DEBUG - post-decoding nbytes 4766910 0.09
2016-04-05 13:50:29,548 - DEBUG - post-decoding nbytes 4817905 0.06
2016-04-05 13:50:43,223 - DEBUG - post-decoding nbytes 3553152 0.08
2016-04-05 13:51:10,810 - DEBUG - post-decoding nbytes 3538892 0.09
2016-04-05 13:51:33,257 - DEBUG - post-decoding nbytes 4019613 0.09
2016-04-05 13:51:48,334 - DEBUG - post-decoding nbytes 3539673 0.08
2016-04-05 13:52:09,617 - DEBUG - post-decoding nbytes 3548588 0.11
2016-04-05 13:52:29,755 - DEBUG - post-decoding nbytes 3627668 0.07
2016-04-05 13:52:46,520 - DEBUG - post-decoding nbytes 3544129 0.11

#### /var/log/spotter_5.err

2016-04-05 13:44:00,474 - DEBUG - post-decoding nbytes 3558312 0.17
2016-04-05 13:44:07,403 - DEBUG - post-decoding nbytes 3555201 0.13
2016-04-05 13:44:18,496 - DEBUG - post-decoding nbytes 3543562 0.07
2016-04-05 13:44:31,832 - DEBUG - post-decoding nbytes 3548587 0.06
2016-04-05 13:45:57,741 - DEBUG - post-decoding nbytes 3548036 58.19
2016-04-05 13:46:06,281 - DEBUG - post-decoding nbytes 3547001 0.06
2016-04-05 13:46:21,258 - DEBUG - post-decoding nbytes 3550381 0.06
2016-04-05 13:46:39,458 - DEBUG - post-decoding nbytes 3551051 0.10
2016-04-05 13:47:03,346 - DEBUG - post-decoding nbytes 3555225 0.11
2016-04-05 13:48:06,748 - DEBUG - post-decoding nbytes 3545768 41.04
2016-04-05 13:48:49,822 - DEBUG - post-decoding nbytes 3562015 23.80
2016-04-05 13:49:12,704 - DEBUG - post-decoding nbytes 4619893 0.13
2016-04-05 13:49:30,712 - DEBUG - post-decoding nbytes 3691173 0.08
2016-04-05 13:49:55,917 - DEBUG - post-decoding nbytes 4160102 0.14
2016-04-05 13:50:07,078 - DEBUG - post-decoding nbytes 3565703 0.05
2016-04-05 13:50:23,191 - DEBUG - post-decoding nbytes 3563725 0.09
2016-04-05 13:50:48,560 - DEBUG - post-decoding nbytes 3561365 0.07
2016-04-05 13:51:16,041 - DEBUG - post-decoding nbytes 4946657 0.06
2016-04-05 13:51:27,516 - DEBUG - post-decoding nbytes 4721926 0.08
2016-04-05 13:51:41,409 - DEBUG - post-decoding nbytes 3551531 0.10
2016-04-05 13:52:00,415 - DEBUG - post-decoding nbytes 3569501 0.12
2016-04-05 13:52:20,846 - DEBUG - post-decoding nbytes 4932391 0.09
2016-04-05 13:52:32,088 - DEBUG - post-decoding nbytes 4852686 0.06
2016-04-05 13:52:42,476 - DEBUG - post-decoding nbytes 3562156 0.06

#### /var/log/spotter_6.err

2016-04-05 13:44:00,774 - DEBUG - post-decoding nbytes 3543869 0.25
2016-04-05 13:44:08,518 - DEBUG - post-decoding nbytes 3546561 0.19
2016-04-05 13:44:23,563 - DEBUG - post-decoding nbytes 3560531 0.07
2016-04-05 13:44:38,216 - DEBUG - post-decoding nbytes 3554889 0.05
2016-04-05 13:45:08,259 - DEBUG - post-decoding nbytes 3560565 0.09
2016-04-05 13:45:32,680 - DEBUG - post-decoding nbytes 3566337 0.07
2016-04-05 13:45:53,244 - DEBUG - post-decoding nbytes 3559177 0.06
2016-04-05 13:46:02,037 - DEBUG - post-decoding nbytes 3543363 0.04
2016-04-05 13:46:10,053 - DEBUG - post-decoding nbytes 3578884 0.08
2016-04-05 13:46:30,487 - DEBUG - post-decoding nbytes 3564037 0.08
2016-04-05 13:46:51,594 - DEBUG - post-decoding nbytes 3637721 0.05
2016-04-05 13:47:15,064 - DEBUG - post-decoding nbytes 3545495 0.09
2016-04-05 13:47:37,953 - DEBUG - post-decoding nbytes 3551357 0.10
2016-04-05 13:47:58,706 - DEBUG - post-decoding nbytes 3554349 0.08
2016-04-05 13:48:16,032 - DEBUG - post-decoding nbytes 3545894 0.07
2016-04-05 13:48:38,669 - DEBUG - post-decoding nbytes 3553345 0.12
2016-04-05 13:50:07,877 - DEBUG - post-decoding nbytes 4466840 64.38
2016-04-05 13:50:16,266 - DEBUG - post-decoding nbytes 4152600 0.06
2016-04-05 13:50:35,799 - DEBUG - post-decoding nbytes 3541285 0.12
2016-04-05 13:51:01,368 - DEBUG - post-decoding nbytes 3551311 0.06
2016-04-05 13:52:01,815 - DEBUG - post-decoding nbytes 3551265 34.08
2016-04-05 13:52:22,410 - DEBUG - post-decoding nbytes 4931690 0.12
2016-04-05 13:52:32,901 - DEBUG - post-decoding nbytes 5001901 0.06
2016-04-05 13:52:43,302 - DEBUG - post-decoding nbytes 3560799 0.07

#### /var/log/spotter_7.err

2016-04-05 13:44:00,754 - DEBUG - post-decoding nbytes 3552104 0.37
2016-04-05 13:44:07,160 - DEBUG - post-decoding nbytes 3549421 0.10
2016-04-05 13:44:21,810 - DEBUG - post-decoding nbytes 3552983 0.10
2016-04-05 13:44:36,429 - DEBUG - post-decoding nbytes 3557410 0.11
2016-04-05 13:45:04,570 - DEBUG - post-decoding nbytes 3547036 0.12
2016-04-05 13:45:29,293 - DEBUG - post-decoding nbytes 3558123 0.12
2016-04-05 13:45:51,757 - DEBUG - post-decoding nbytes 3550837 0.06
2016-04-05 13:46:12,790 - DEBUG - post-decoding nbytes 3542517 11.74
2016-04-05 13:46:53,753 - DEBUG - post-decoding nbytes 3535268 22.60
2016-04-05 13:47:18,981 - DEBUG - post-decoding nbytes 3564525 0.10
2016-04-05 13:47:40,296 - DEBUG - post-decoding nbytes 3731502 0.11
2016-04-05 13:47:59,443 - DEBUG - post-decoding nbytes 3556630 0.06
2016-04-05 13:48:16,516 - DEBUG - post-decoding nbytes 3548969 0.10
2016-04-05 13:48:40,270 - DEBUG - post-decoding nbytes 3550773 0.08
2016-04-05 13:49:03,331 - DEBUG - post-decoding nbytes 3627468 0.04
2016-04-05 13:49:28,198 - DEBUG - post-decoding nbytes 3559328 0.08
2016-04-05 13:49:57,116 - DEBUG - post-decoding nbytes 3732635 0.05
2016-04-05 13:51:20,176 - DEBUG - post-decoding nbytes 3556280 70.81
2016-04-05 13:51:40,735 - DEBUG - post-decoding nbytes 3555175 0.04
2016-04-05 13:52:00,492 - DEBUG - post-decoding nbytes 3559843 0.07
2016-04-05 13:52:56,514 - DEBUG - post-decoding nbytes 4569966 35.33

#### /var/log/spotter_8.err

2016-04-05 13:44:01,011 - DEBUG - post-decoding nbytes 3549601 0.42
2016-04-05 13:44:07,835 - DEBUG - post-decoding nbytes 3548996 0.25
2016-04-05 13:44:21,909 - DEBUG - post-decoding nbytes 3548206 0.15
2016-04-05 13:45:03,520 - DEBUG - post-decoding nbytes 3556967 30.91
2016-04-05 13:45:59,873 - DEBUG - post-decoding nbytes 3546452 33.39
2016-04-05 13:46:06,641 - DEBUG - post-decoding nbytes 3550975 0.05
2016-04-05 13:46:24,236 - DEBUG - post-decoding nbytes 3564972 0.06
2016-04-05 13:46:40,274 - DEBUG - post-decoding nbytes 3547466 0.11
2016-04-05 13:47:02,863 - DEBUG - post-decoding nbytes 3557632 0.10
2016-04-05 13:47:59,982 - DEBUG - post-decoding nbytes 3594544 33.60
2016-04-05 13:48:17,571 - DEBUG - post-decoding nbytes 3549917 0.10
2016-04-05 13:48:42,989 - DEBUG - post-decoding nbytes 4401808 0.14
2016-04-05 13:48:58,935 - DEBUG - post-decoding nbytes 3560086 0.06
2016-04-05 13:49:22,134 - DEBUG - post-decoding nbytes 3551868 0.05
2016-04-05 13:49:53,667 - DEBUG - post-decoding nbytes 3540897 0.11
2016-04-05 13:50:52,003 - DEBUG - post-decoding nbytes 3540393 40.68
2016-04-05 13:51:19,727 - DEBUG - post-decoding nbytes 4495092 0.11
2016-04-05 13:51:33,168 - DEBUG - post-decoding nbytes 3733953 0.11
2016-04-05 13:51:48,519 - DEBUG - post-decoding nbytes 3569533 0.04
2016-04-05 13:52:10,624 - DEBUG - post-decoding nbytes 4026741 0.10
2016-04-05 13:52:43,675 - DEBUG - post-decoding nbytes 3545872 18.22

#### /var/log/spotter_9.err

2016-04-05 13:44:00,849 - DEBUG - post-decoding nbytes 3556060 0.39
2016-04-05 13:44:07,296 - DEBUG - post-decoding nbytes 3550145 0.10
2016-04-05 13:44:20,703 - DEBUG - post-decoding nbytes 3552083 0.07
2016-04-05 13:44:34,021 - DEBUG - post-decoding nbytes 3554712 0.06
2016-04-05 13:45:02,630 - DEBUG - post-decoding nbytes 3554325 0.14
2016-04-05 13:45:29,030 - DEBUG - post-decoding nbytes 3553436 0.06
2016-04-05 13:45:52,168 - DEBUG - post-decoding nbytes 3556318 0.07
2016-04-05 13:46:01,484 - DEBUG - post-decoding nbytes 3541635 0.06
2016-04-05 13:46:11,133 - DEBUG - post-decoding nbytes 3550155 0.09
2016-04-05 13:46:30,201 - DEBUG - post-decoding nbytes 3549773 0.05
2016-04-05 13:46:47,790 - DEBUG - post-decoding nbytes 3971319 0.05
2016-04-05 13:47:08,486 - DEBUG - post-decoding nbytes 3664392 0.09
2016-04-05 13:47:28,072 - DEBUG - post-decoding nbytes 3573766 0.11
2016-04-05 13:47:51,483 - DEBUG - post-decoding nbytes 3557813 0.07
2016-04-05 13:48:56,566 - DEBUG - post-decoding nbytes 4092917 44.74
2016-04-05 13:49:10,811 - DEBUG - post-decoding nbytes 3958427 0.08
2016-04-05 13:50:08,858 - DEBUG - post-decoding nbytes 3555206 34.06
2016-04-05 13:50:28,333 - DEBUG - post-decoding nbytes 3561262 0.10
2016-04-05 13:51:39,669 - DEBUG - post-decoding nbytes 4948418 46.44
2016-04-05 13:51:48,746 - DEBUG - post-decoding nbytes 3545728 0.14
2016-04-05 13:52:09,218 - DEBUG - post-decoding nbytes 3563773 0.13

#### /var/log/spotter_10.err

2016-04-05 13:44:00,948 - DEBUG - post-decoding nbytes 3559236 0.37
2016-04-05 13:44:08,271 - DEBUG - post-decoding nbytes 3547580 0.24
2016-04-05 13:44:23,229 - DEBUG - post-decoding nbytes 3553133 0.06
2016-04-05 13:44:34,456 - DEBUG - post-decoding nbytes 3554431 0.07
2016-04-05 13:45:42,613 - DEBUG - post-decoding nbytes 3567379 43.44
2016-04-05 13:45:58,024 - DEBUG - post-decoding nbytes 3556014 0.10
2016-04-05 13:46:05,693 - DEBUG - post-decoding nbytes 3553535 0.08
2016-04-05 13:46:21,785 - DEBUG - post-decoding nbytes 3560001 0.06
2016-04-05 13:46:42,876 - DEBUG - post-decoding nbytes 3546689 0.07
2016-04-05 13:47:06,479 - DEBUG - post-decoding nbytes 3550362 0.12
2016-04-05 13:47:29,372 - DEBUG - post-decoding nbytes 3555182 0.09
2016-04-05 13:48:28,302 - DEBUG - post-decoding nbytes 3543561 36.48
2016-04-05 13:48:54,630 - DEBUG - post-decoding nbytes 4500619 0.07
2016-04-05 13:49:05,815 - DEBUG - post-decoding nbytes 4320741 0.09
2016-04-05 13:49:23,259 - DEBUG - post-decoding nbytes 3551493 0.06
2016-04-05 13:49:53,452 - DEBUG - post-decoding nbytes 3534454 0.11
2016-04-05 13:51:11,705 - DEBUG - post-decoding nbytes 3560159 62.25
2016-04-05 13:51:35,260 - DEBUG - post-decoding nbytes 3962749 0.12
2016-04-05 13:51:48,761 - DEBUG - post-decoding nbytes 4828065 0.17
2016-04-05 13:52:01,345 - DEBUG - post-decoding nbytes 3554465 0.08
2016-04-05 13:52:21,458 - DEBUG - post-decoding nbytes 3554681 0.11
2016-04-05 13:52:40,368 - DEBUG - post-decoding nbytes 3550998 0.04

#### /var/log/spotter_11.err

2016-04-05 13:44:00,500 - DEBUG - post-decoding nbytes 3544720 0.19
2016-04-05 13:44:07,533 - DEBUG - post-decoding nbytes 3556079 0.13
2016-04-05 13:45:09,554 - DEBUG - post-decoding nbytes 3546633 51.44
2016-04-05 13:45:35,431 - DEBUG - post-decoding nbytes 3551445 0.06
2016-04-05 13:45:55,755 - DEBUG - post-decoding nbytes 3554908 0.10
2016-04-05 13:46:03,738 - DEBUG - post-decoding nbytes 3556872 0.10
2016-04-05 13:46:16,811 - DEBUG - post-decoding nbytes 3573794 0.08
2016-04-05 13:46:34,653 - DEBUG - post-decoding nbytes 3548072 0.07
2016-04-05 13:46:52,602 - DEBUG - post-decoding nbytes 3565726 0.09
2016-04-05 13:47:16,973 - DEBUG - post-decoding nbytes 3551052 0.03
2016-04-05 13:47:41,438 - DEBUG - post-decoding nbytes 3553507 0.11
2016-04-05 13:48:00,220 - DEBUG - post-decoding nbytes 3562329 0.05
2016-04-05 13:48:17,434 - DEBUG - post-decoding nbytes 3552523 0.11
2016-04-05 13:48:40,360 - DEBUG - post-decoding nbytes 3541337 0.13
2016-04-05 13:49:04,443 - DEBUG - post-decoding nbytes 3572806 0.12
2016-04-05 13:49:32,387 - DEBUG - post-decoding nbytes 3563775 0.07
2016-04-05 13:49:59,943 - DEBUG - post-decoding nbytes 3577953 0.07
2016-04-05 13:50:12,149 - DEBUG - post-decoding nbytes 3557453 0.09
2016-04-05 13:51:45,906 - DEBUG - post-decoding nbytes 3550458 69.09
2016-04-05 13:52:06,506 - DEBUG - post-decoding nbytes 3561012 0.09
2016-04-05 13:52:27,164 - DEBUG - post-decoding nbytes 3549888 0.06

#### /var/log/spotter_12.err

2016-04-05 13:44:00,713 - DEBUG - post-decoding nbytes 3554697 0.33
2016-04-05 13:44:07,178 - DEBUG - post-decoding nbytes 3553100 0.11
2016-04-05 13:44:18,769 - DEBUG - post-decoding nbytes 3556598 0.11
2016-04-05 13:44:31,559 - DEBUG - post-decoding nbytes 3560644 0.05
2016-04-05 13:45:32,138 - DEBUG - post-decoding nbytes 3551469 39.15
2016-04-05 13:45:53,376 - DEBUG - post-decoding nbytes 3543263 0.06
2016-04-05 13:46:02,427 - DEBUG - post-decoding nbytes 3550090 0.06
2016-04-05 13:46:08,597 - DEBUG - post-decoding nbytes 3558817 0.02
2016-04-05 13:46:26,370 - DEBUG - post-decoding nbytes 3551219 0.05
2016-04-05 13:46:42,575 - DEBUG - post-decoding nbytes 3568348 0.08
2016-04-05 13:47:06,132 - DEBUG - post-decoding nbytes 3547601 0.13
2016-04-05 13:47:26,986 - DEBUG - post-decoding nbytes 3728118 0.06
2016-04-05 13:47:46,627 - DEBUG - post-decoding nbytes 3560069 0.05
2016-04-05 13:48:07,371 - DEBUG - post-decoding nbytes 4688188 0.11
2016-04-05 13:48:18,615 - DEBUG - post-decoding nbytes 3544882 0.03
2016-04-05 13:48:46,186 - DEBUG - post-decoding nbytes 3559195 0.09
2016-04-05 13:49:09,028 - DEBUG - post-decoding nbytes 3559124 0.09
2016-04-05 13:49:38,930 - DEBUG - post-decoding nbytes 3568437 0.12
2016-04-05 13:50:03,571 - DEBUG - post-decoding nbytes 3549179 0.07
2016-04-05 13:50:57,712 - DEBUG - post-decoding nbytes 4510263 40.29
2016-04-05 13:51:15,129 - DEBUG - post-decoding nbytes 3562132 0.06
2016-04-05 13:51:36,671 - DEBUG - post-decoding nbytes 3575504 0.13
2016-04-05 13:51:56,296 - DEBUG - post-decoding nbytes 3552461 0.09
2016-04-05 13:52:17,036 - DEBUG - post-decoding nbytes 3566593 0.10
2016-04-05 13:52:35,630 - DEBUG - post-decoding nbytes 3558270 0.08
2016-04-05 13:52:56,302 - DEBUG - post-decoding nbytes 4224022 0.09

#### /var/log/spotter_13.err

2016-04-05 13:44:00,982 - DEBUG - post-decoding nbytes 3564472 0.40
2016-04-05 13:44:07,738 - DEBUG - post-decoding nbytes 3549498 0.37
2016-04-05 13:44:23,431 - DEBUG - post-decoding nbytes 3551839 0.07
2016-04-05 13:45:49,324 - DEBUG - post-decoding nbytes 3554567 73.66
2016-04-05 13:46:01,122 - DEBUG - post-decoding nbytes 3544290 0.06
2016-04-05 13:47:08,711 - DEBUG - post-decoding nbytes 3556544 58.50
2016-04-05 13:47:30,938 - DEBUG - post-decoding nbytes 3567703 0.05
2016-04-05 13:47:53,106 - DEBUG - post-decoding nbytes 3588123 0.05
2016-04-05 13:48:10,270 - DEBUG - post-decoding nbytes 3546058 0.05
2016-04-05 13:49:49,088 - DEBUG - post-decoding nbytes 3551481 78.39
2016-04-05 13:50:08,026 - DEBUG - post-decoding nbytes 4693379 0.09
2016-04-05 13:50:15,510 - DEBUG - post-decoding nbytes 4879540 0.10
2016-04-05 13:50:30,331 - DEBUG - post-decoding nbytes 4932582 0.12
2016-04-05 13:50:44,624 - DEBUG - post-decoding nbytes 3549785 0.08
2016-04-05 13:51:12,716 - DEBUG - post-decoding nbytes 3552006 0.05
2016-04-05 13:52:27,173 - DEBUG - post-decoding nbytes 4458770 52.26

#### /var/log/spotter_14.err

2016-04-05 13:44:00,377 - DEBUG - post-decoding nbytes 3563760 0.15
2016-04-05 13:44:07,397 - DEBUG - post-decoding nbytes 3543031 0.12
2016-04-05 13:44:21,559 - DEBUG - post-decoding nbytes 3553640 0.13
2016-04-05 13:44:36,293 - DEBUG - post-decoding nbytes 3551698 0.16
2016-04-05 13:45:05,477 - DEBUG - post-decoding nbytes 3551468 0.16
2016-04-05 13:45:56,550 - DEBUG - post-decoding nbytes 3553809 26.58
2016-04-05 13:46:04,169 - DEBUG - post-decoding nbytes 3565290 0.05
2016-04-05 13:46:15,780 - DEBUG - post-decoding nbytes 3545379 0.04
2016-04-05 13:46:34,712 - DEBUG - post-decoding nbytes 3565523 0.05
2016-04-05 13:46:54,902 - DEBUG - post-decoding nbytes 3559801 0.12
2016-04-05 13:47:19,625 - DEBUG - post-decoding nbytes 3548494 0.03
2016-04-05 13:48:04,264 - DEBUG - post-decoding nbytes 3548689 22.50
2016-04-05 13:48:22,928 - DEBUG - post-decoding nbytes 3573259 0.07
2016-04-05 13:48:50,095 - DEBUG - post-decoding nbytes 3552236 0.09
2016-04-05 13:50:09,118 - DEBUG - post-decoding nbytes 3562924 57.85
2016-04-05 13:50:29,501 - DEBUG - post-decoding nbytes 3568944 0.11
2016-04-05 13:50:55,438 - DEBUG - post-decoding nbytes 3546981 0.06
2016-04-05 13:51:22,658 - DEBUG - post-decoding nbytes 4889211 0.10
2016-04-05 13:51:33,641 - DEBUG - post-decoding nbytes 3557173 0.07
2016-04-05 13:51:52,549 - DEBUG - post-decoding nbytes 3560149 0.05
2016-04-05 13:52:13,111 - DEBUG - post-decoding nbytes 4898391 0.06
2016-04-05 13:52:24,274 - DEBUG - post-decoding nbytes 3558418 0.05

#### /var/log/spotter_15.err

2016-04-05 13:44:00,755 - DEBUG - post-decoding nbytes 3541006 0.29
2016-04-05 13:44:07,359 - DEBUG - post-decoding nbytes 3547327 0.12
2016-04-05 13:44:19,263 - DEBUG - post-decoding nbytes 3551192 0.05
2016-04-05 13:44:32,971 - DEBUG - post-decoding nbytes 3546776 0.05
2016-04-05 13:45:00,850 - DEBUG - post-decoding nbytes 3558257 0.07
2016-04-05 13:45:26,988 - DEBUG - post-decoding nbytes 3551233 0.04
2016-04-05 13:45:49,809 - DEBUG - post-decoding nbytes 3560887 0.05
2016-04-05 13:46:00,265 - DEBUG - post-decoding nbytes 3545002 0.03
2016-04-05 13:46:09,170 - DEBUG - post-decoding nbytes 3555526 0.03
2016-04-05 13:46:27,323 - DEBUG - post-decoding nbytes 3555946 0.06
2016-04-05 13:46:47,005 - DEBUG - post-decoding nbytes 3557793 0.08
2016-04-05 13:47:11,237 - DEBUG - post-decoding nbytes 3539232 0.06
2016-04-05 13:47:34,201 - DEBUG - post-decoding nbytes 4457957 0.07
2016-04-05 13:47:48,905 - DEBUG - post-decoding nbytes 3728994 0.08
2016-04-05 13:48:05,893 - DEBUG - post-decoding nbytes 3556229 0.04
2016-04-05 13:48:25,410 - DEBUG - post-decoding nbytes 3546790 0.06
2016-04-05 13:48:51,581 - DEBUG - post-decoding nbytes 3568583 0.14
2016-04-05 13:49:11,709 - DEBUG - post-decoding nbytes 3560919 0.05
2016-04-05 13:49:42,019 - DEBUG - post-decoding nbytes 4605677 0.07
2016-04-05 13:49:58,246 - DEBUG - post-decoding nbytes 4912591 0.07
2016-04-05 13:50:06,805 - DEBUG - post-decoding nbytes 3563678 0.03
2016-04-05 13:50:23,556 - DEBUG - post-decoding nbytes 3553809 0.07
2016-04-05 13:50:47,915 - DEBUG - post-decoding nbytes 3573548 0.06
2016-04-05 13:51:14,232 - DEBUG - post-decoding nbytes 4973020 0.06
2016-04-05 13:51:56,998 - DEBUG - post-decoding nbytes 4900227 29.36
2016-04-05 13:52:08,321 - DEBUG - post-decoding nbytes 3555993 0.07
2016-04-05 13:52:28,090 - DEBUG - post-decoding nbytes 3562488 0.15
2016-04-05 13:52:48,091 - DEBUG - post-decoding nbytes 4760390 0.09

#### /var/log/spotter_16.err

2016-04-05 13:44:01,090 - DEBUG - post-decoding nbytes 3549826 0.35
2016-04-05 13:44:53,520 - DEBUG - post-decoding nbytes 3544873 44.04
2016-04-05 13:45:19,244 - DEBUG - post-decoding nbytes 3556091 0.06
2016-04-05 13:45:42,021 - DEBUG - post-decoding nbytes 3564080 0.07
2016-04-05 13:45:57,895 - DEBUG - post-decoding nbytes 3552492 0.05
2016-04-05 13:46:07,676 - DEBUG - post-decoding nbytes 3542451 0.04
2016-04-05 13:46:27,037 - DEBUG - post-decoding nbytes 3551091 0.07
2016-04-05 13:46:46,230 - DEBUG - post-decoding nbytes 3549293 0.04
2016-04-05 13:47:11,517 - DEBUG - post-decoding nbytes 3546542 0.09
2016-04-05 13:47:35,368 - DEBUG - post-decoding nbytes 4158172 0.12
2016-04-05 13:47:50,777 - DEBUG - post-decoding nbytes 3601769 0.08
2016-04-05 13:48:09,218 - DEBUG - post-decoding nbytes 3560034 0.06
2016-04-05 13:48:30,943 - DEBUG - post-decoding nbytes 3535348 0.06
2016-04-05 13:48:57,046 - DEBUG - post-decoding nbytes 4634842 0.09
2016-04-05 13:49:58,330 - DEBUG - post-decoding nbytes 4527174 51.03
2016-04-05 13:50:07,558 - DEBUG - post-decoding nbytes 3555623 0.08
2016-04-05 13:50:24,959 - DEBUG - post-decoding nbytes 3551161 0.04
2016-04-05 13:51:34,767 - DEBUG - post-decoding nbytes 4076136 45.31
2016-04-05 13:52:40,516 - DEBUG - post-decoding nbytes 3556508 53.86

#### /var/log/spotter_17.err

2016-04-05 13:44:01,221 - DEBUG - post-decoding nbytes 3543076 0.55
2016-04-05 13:44:07,208 - DEBUG - post-decoding nbytes 3541935 0.08
2016-04-05 13:44:20,106 - DEBUG - post-decoding nbytes 3545816 0.09
2016-04-05 13:44:35,587 - DEBUG - post-decoding nbytes 3551615 0.13
2016-04-05 13:45:02,650 - DEBUG - post-decoding nbytes 3557234 0.19
2016-04-05 13:45:57,735 - DEBUG - post-decoding nbytes 3556616 30.06
2016-04-05 13:46:05,001 - DEBUG - post-decoding nbytes 3558342 0.05
2016-04-05 13:46:20,040 - DEBUG - post-decoding nbytes 3546555 0.04
2016-04-05 13:47:20,412 - DEBUG - post-decoding nbytes 3553934 41.82
2016-04-05 13:47:43,444 - DEBUG - post-decoding nbytes 3555478 0.09
2016-04-05 13:49:07,576 - DEBUG - post-decoding nbytes 3546131 63.00
2016-04-05 13:49:37,394 - DEBUG - post-decoding nbytes 3555834 0.09
2016-04-05 13:50:03,253 - DEBUG - post-decoding nbytes 3547475 0.04
2016-04-05 13:50:18,279 - DEBUG - post-decoding nbytes 3557821 0.05
2016-04-05 13:50:43,651 - DEBUG - post-decoding nbytes 3559647 0.12
2016-04-05 13:51:10,567 - DEBUG - post-decoding nbytes 3711661 0.11
2016-04-05 13:51:31,477 - DEBUG - post-decoding nbytes 3560098 0.03
2016-04-05 13:52:22,054 - DEBUG - post-decoding nbytes 4972657 32.52
2016-04-05 13:52:32,627 - DEBUG - post-decoding nbytes 3966155 0.12
2016-04-05 13:52:46,340 - DEBUG - post-decoding nbytes 3552397 0.12

#### /var/log/spotter_18.err

2016-04-05 13:44:00,966 - DEBUG - post-decoding nbytes 3552529 0.30
2016-04-05 13:44:08,226 - DEBUG - post-decoding nbytes 3549135 0.15
2016-04-05 13:44:21,429 - DEBUG - post-decoding nbytes 3561462 0.07
2016-04-05 13:44:36,286 - DEBUG - post-decoding nbytes 3544588 0.07
2016-04-05 13:45:05,135 - DEBUG - post-decoding nbytes 3550215 0.08
2016-04-05 13:45:29,616 - DEBUG - post-decoding nbytes 3554517 0.09
2016-04-05 13:45:51,894 - DEBUG - post-decoding nbytes 3559867 0.08
2016-04-05 13:46:01,632 - DEBUG - post-decoding nbytes 3556493 0.05
2016-04-05 13:46:10,813 - DEBUG - post-decoding nbytes 3542682 0.13
2016-04-05 13:46:29,911 - DEBUG - post-decoding nbytes 3555445 0.07
2016-04-05 13:47:31,317 - DEBUG - post-decoding nbytes 3886160 42.18
2016-04-05 13:47:49,593 - DEBUG - post-decoding nbytes 3988774 0.11
2016-04-05 13:48:03,705 - DEBUG - post-decoding nbytes 3555301 0.06
2016-04-05 13:49:01,984 - DEBUG - post-decoding nbytes 3569969 41.06
2016-04-05 13:50:05,847 - DEBUG - post-decoding nbytes 3670112 39.63
2016-04-05 13:51:29,420 - DEBUG - post-decoding nbytes 3556707 70.99
2016-04-05 13:51:47,459 - DEBUG - post-decoding nbytes 3558695 0.09
2016-04-05 13:52:08,967 - DEBUG - post-decoding nbytes 3577233 0.05
2016-04-05 13:52:29,574 - DEBUG - post-decoding nbytes 3561534 0.04
2016-04-05 13:52:49,869 - DEBUG - post-decoding nbytes 3551020 0.06

#### /var/log/spotter_19.err

2016-04-05 13:44:00,303 - DEBUG - post-decoding nbytes 3548768 0.08
2016-04-05 13:44:07,630 - DEBUG - post-decoding nbytes 3552223 0.25
2016-04-05 13:44:21,870 - DEBUG - post-decoding nbytes 3541983 0.11
2016-04-05 13:44:33,805 - DEBUG - post-decoding nbytes 3548998 0.11
2016-04-05 13:44:57,311 - DEBUG - post-decoding nbytes 3549102 0.12
2016-04-05 13:45:22,515 - DEBUG - post-decoding nbytes 3557740 0.07
2016-04-05 13:45:45,698 - DEBUG - post-decoding nbytes 3546356 0.05
2016-04-05 13:45:59,386 - DEBUG - post-decoding nbytes 3552746 0.03
2016-04-05 13:46:07,061 - DEBUG - post-decoding nbytes 3548177 0.05
2016-04-05 13:46:23,800 - DEBUG - post-decoding nbytes 3549090 0.07
2016-04-05 13:46:40,289 - DEBUG - post-decoding nbytes 3557653 0.09
2016-04-05 13:47:02,108 - DEBUG - post-decoding nbytes 3546567 0.13
2016-04-05 13:47:59,115 - DEBUG - post-decoding nbytes 3568436 35.16
2016-04-05 13:48:16,423 - DEBUG - post-decoding nbytes 3552043 0.09
2016-04-05 13:48:40,820 - DEBUG - post-decoding nbytes 4045055 0.07
2016-04-05 13:48:59,571 - DEBUG - post-decoding nbytes 3553056 0.05
2016-04-05 13:49:23,382 - DEBUG - post-decoding nbytes 3549622 0.03
2016-04-05 13:49:54,029 - DEBUG - post-decoding nbytes 3546838 0.13
2016-04-05 13:50:09,750 - DEBUG - post-decoding nbytes 3546356 0.04
2016-04-05 13:50:30,438 - DEBUG - post-decoding nbytes 3629546 0.08
2016-04-05 13:50:52,976 - DEBUG - post-decoding nbytes 3560308 0.07
2016-04-05 13:51:20,638 - DEBUG - post-decoding nbytes 4156712 0.10
2016-04-05 13:51:35,920 - DEBUG - post-decoding nbytes 3627305 0.06
2016-04-05 13:51:53,451 - DEBUG - post-decoding nbytes 3568445 0.05
2016-04-05 13:52:14,738 - DEBUG - post-decoding nbytes 4423719 0.05
2016-04-05 13:52:27,370 - DEBUG - post-decoding nbytes 3553845 0.04
2016-04-05 13:52:45,631 - DEBUG - post-decoding nbytes 3739728 0.09

#### /var/log/spotter_20.err

2016-04-05 13:44:00,652 - DEBUG - post-decoding nbytes 3557078 0.19
2016-04-05 13:44:07,468 - DEBUG - post-decoding nbytes 3557532 0.15
2016-04-05 13:44:18,323 - DEBUG - post-decoding nbytes 3544368 0.09
2016-04-05 13:45:36,734 - DEBUG - post-decoding nbytes 3553396 64.21
2016-04-05 13:45:55,357 - DEBUG - post-decoding nbytes 3540464 0.07
2016-04-05 13:46:02,688 - DEBUG - post-decoding nbytes 3555844 0.04
2016-04-05 13:46:36,455 - DEBUG - post-decoding nbytes 3558639 26.66
2016-04-05 13:48:02,963 - DEBUG - post-decoding nbytes 3558382 64.10
2016-04-05 13:48:22,051 - DEBUG - post-decoding nbytes 3557465 0.08
2016-04-05 13:48:49,330 - DEBUG - post-decoding nbytes 3543067 0.11
2016-04-05 13:49:10,681 - DEBUG - post-decoding nbytes 3672006 0.10
2016-04-05 13:49:39,583 - DEBUG - post-decoding nbytes 3554456 0.09
2016-04-05 13:50:04,070 - DEBUG - post-decoding nbytes 3550699 0.08
2016-04-05 13:50:16,935 - DEBUG - post-decoding nbytes 4782374 0.17
2016-04-05 13:50:32,666 - DEBUG - post-decoding nbytes 3562285 0.05
2016-04-05 13:51:40,188 - DEBUG - post-decoding nbytes 4112708 40.87
2016-04-05 13:52:25,036 - DEBUG - post-decoding nbytes 4763418 32.71
2016-04-05 13:52:36,245 - DEBUG - post-decoding nbytes 3565623 0.06
2016-04-05 13:52:56,734 - DEBUG - post-decoding nbytes 3555180 0.10
[
simongregorebner commented 8 years ago

One other remark I would like to place on this pull-request I would not explicitly check for Mac OS X and disable OpenMP as there are also OpenMP implementations/support for Mac OS X (gcc and clang (e.g. https://clang-omp.github.io/))

Beside that, it is counter intuitive to set the OPENMP_ENABLED flag to true and still not ending up with an OpenMP enabled code/or errors caused because I have no OpenMP installed =)

leosala commented 8 years ago

uhm, on a second thought I'd also prefer OpenMP switched off by default, as this library is likely to be included in bigger projects / pipelines, and OpenMP can interact in an unexpected way. As @simongregorebner , I will leave the final decision to you.

One thing I'm gonna do is to create another commit, where the behavior of OPENMP_ENABLED can be driven by a env variable (this should ease also packaging e.g. with conda)

leosala commented 8 years ago

I put again OPENMP_ENABLED False by default (see discussion above), let me know if you I should put back to True again... thanks!

I've also removed the explicit MacOsX check, given comment by @simongregorebner

kiyo-masui commented 8 years ago

Okay, I've given this some thought and the proper way to do this is with a command line argument to build_ext, which could also be supplied in a setup.cfg. This means subclassing the current build_ext to accept the argument, in much the same way that install is currently subclassed. The default value of the argument could be compiler dependant, but that is a bit more work.

This would be pretty quick for me to implement, but it will take me a few days to get around to it. If anyone else wants to take a crack at it, they are welcome.

Incidentally, @simongregorebner's timings are troubling. If you could file an issue with a minimal example I can look into fixing the log jam.