NOTE: This file is best viewed at https://github.com/ORNL-Fusion/kineticj/
The Kinetic-j program takes a time harmonic electric wave field, and calculates the kinetic plasma current. This calculation is done in configuration-space, as opposed to the more traditional Fourier-space evaluation forms provided in the standard plasma wave physics texts.
The program is organized into C++ and IDL files. The C++ are the buildable program (located in src and include directories), with the IDL scripts (located in the idl directory) being used to generate inputs, run benchmarks, and post process outputs. The bin and obj directories are empty, being only placeholders for build locations. The mathematica directory contains some relevant analysis and derivations of the equations upon which the program is based.
. (Makefile etc)
├── benchmarks (template and output for the IDL based sigma benchmarking)
├── bin (empty, but where the binary ends up)
├── idl (IDL utility scripts)
├── include (C++ header files)
├── machine-makefiles (machine specific makefile setting files)
├── mathematica (mathematica scripts for proof of principle and Z-function table creation)
├── matlab (matlab utility scripts)
├── obj (empty)
├── python (python scripts for plotting and testing)
├── src (C++ source files)
├── template (single test case)
└── tests (regression test cases with reference solutions)
The human readable input files are of the format defined by the libconfig API. Libconfig can be found at the above location on github.
We utilize the netcdf file format for both inputs and outputs.
We further utilize the more recent CXX-4 components of the NetCDF API (note that the --enable-cxx4 flag is not required as suggested in the netcdf-cxx4 github readme file).
The THRUST headers are required and are available either standalone from the THRUST repo (for building on systems without GPUs), or from within the CUDA SDK (for building on systems with GPUs). So if you do not have an nvidia compute GPU, you will still need the THRUST library (unbuilt).
We utilize a simple machine specific makefile to specify the locations of the above dependencies. To build Kinetic-j on a new machine you will need to create a copy of an existing file (e.g., machine-makefiles/Makefile.dlg-macpro
) with the name of your machine as the extension. This can be done via
cp machine-makefiles/Makefile.dlg-macpro machine-makefiles/$(uname -n)
and then editing the resulting file appropriately. Then just
make clean
make
make test
The THRUST API provides for single thread CPU (CPP), multi thread OpenMP (OMP), and CUDA (CUDA) targets via the appropriate choice of THRUST policy. Within the top level Makefile
the THRUST policy (which selects either CPU, OPENMP, or GPU) is selected by uncommenting the desired choice as
# Select only one.
USE_SERIAL:= 0
USE_CUDA := 0
USE_OPENMP:= 1
Also, various code features and debugging levels can be enabled at build time by enabling any of the following within the top level Makefile
.
# Features
CPPFLAGS += -DCYLINDRICAL_INPUT_FIELDS=1 # Are the input (NetCDF) fields in Cylindrical or not.
CPPFLAGS += -D_PARTICLE_BOUNDARY=1 # 1 = particle absorbing walls, 2 = periodic, 3 = reflective
CPPFLAGS += -DGC_ORBITS=0 # Use Guiding Center evalulation (Not yet functional)
CPPFLAGS += -DCOMPLEX_WRF=0 # Use the complex omega as the time integral decay time, as opposed to the Hanning window.
# Non THRUST Implementations
CPPFLAGS += -DDO_CPU_ITERATOR_APPROACH=0
CPPFLAGS += -DDO_CPU_APPROACH=0
# Timing Info
CPPFLAGS += -DUSEPAPI=0
CPPFLAGS += -DLOWMEM_USEPAPI=0
CPPFLAGS += -DCLOCK=1
CPPFLAGS += -DPRINT_INFO=1
# Debug Info Level
CPPFLAGS += -DDEBUG_GC=0
CPPFLAGS += -DDEBUG_EVAL_VGC=0
CPPFLAGS += -DDEBUG_EVAL_APAR=0
CPPFLAGS += -DDEBUGLEVEL=0
CPPFLAGS += -DDEBUG_LINES=0
CPPFLAGS += -DDEBUG_INTERP=0
CPPFLAGS += -DDEBUG_MAXWELLIAN=0
CPPFLAGS += -DDEBUG_FORCE_TERM=0
CPPFLAGS += -DDEBUG_MOVE=0
CPPFLAGS += -DDEBUG_ROTATION=0
CPPFLAGS += -DDEBUG_INTVECARRAY=0
CPPFLAGS += -DDEBUG_READ_E_FIELD=0
# File Write Options
CPPFLAGS += -DLOWMEM_ORBIT_WRITE=0
CPPFLAGS += -DF1_WRITE=0
source env-edison.sh
make clean
make
make test # (although you're better off running the tests on the compute nodes as described below)
source env-gpufusion.sh
make clean
make
make test
Set a KINETICJ_ROOT
environment variable to be the location of the cloned source ...
cd ~/code
git clone https://github.com/ORNL-Fusion/kineticj.git
export KINETICJ_ROOT=~/code/kineticj
To aid development testing we include some simple regression testing. This is run as follows (or via make test
) ...
cd $KINETICJ_ROOT/tests
python ../python/kj_test.py
or at NERSC ...
cd $KINETICJ_ROOT
source env-edison.sh
cd $SCRATCH
mkdir kineticj
cp -r $KINETICJ_ROOT/tests .
cd tests
salloc -N 1 -p debug
python $KINETICJ_ROOT/python/kj_test.py
exit
with the expected output being ...
python $KINETICJ_ROOT/python/kj_test.py
benchmark1-00007 PASS [2.9 seconds]
benchmark2-00013 PASS [2.5 seconds]
benchmark3-00004 PASS [4.2 seconds]
test4 PASS [3.4 seconds]
A standalone test case is also availble where we demonstrate the current response for a variable magnetic field (1/r) where the fundamental and 2nd harmonic ion cyclotron resonances are in the domain (at x=1.75 and x=3.5 respectively). We have provided an input file template/input/input-data.nc
with an electric field with kx=pi/(2*rho_L)
where rho_L
is the Lamor radius at the location of the 2nd harmonic resonance, i.e., the finite Lamor radius interaction is captured per pg.270-271 of Stix. This case is run via the following commands ...
cd $KINETICJ_ROOT
cd template
./bin/kineticj
python ../python/kj_plot.py
or for NERSC (Edison with a single node using OpenMP)
cd $KINETICJ_ROOT
source env-edison.sh
cd $SCRATCH
mkdir kineticj
cp -r $KINETICJ_ROOT/template .
cd template
salloc -N 1 -p debug
$KINETICJ_ROOT/bin/kineticj
exit
python $KINETICJ_ROOT/python/kj_plot.py
Changing the variables in template/kj.cfg
will allow experimenting with running the code.
1 xGridMin = 1.0;
2 xGridMax = 5.0;
3 nXGrid = 200
4 nRFCycles = 50.0;
5 species_number = 0;
6 species_amu = 1.0;
7 species_Z = 1.0;
8 runIdent = "template";
9 nP_Vx = 5;
10 nP_Vy = 5;
11 nP_Vz = 5;
12 nThermal = 3;
13 nPhi = 0;
14 ky = 0.0;
15 T_keV = 5.0;
16 input_fName = "input/input-data.nc";
17 kz = 0.0;
18 nStepsPerCyclotronPeriod = 30.0;
kineticj
and dlg_utils
idl folders to the IDL path, i.e., edit the export $IDL_STARTUP=~/idlStartup.pro
file, e.g.,
!PATH=EXPAND_PATH('+~/code/kineticj/idl:+~/code/dlg_utils/idl:<IDL_DEFAULT>')
cd $KINETICJ_ROOT
cd benchmarks
./clean.sh
idl
IDL>kj_sigma_benchmarks, runKJ=1, benchmark=1
or at NERSC (Edison with a single node using OpenMP)
cd $KINETICJ_ROOT
source env-edison.sh
cd $SCRATCH
mkdir kineticj
cp -r $KINETICJ_ROOT/benchmarks .
cd benchmarks
./clean.sh
salloc -N 1 -p debug
idl
IDL>kj_sigma_benchmarks, runKJ=1, benchmark=1
kineticj/benchmarks/benchmark1/benchmark.png
file for output. To speed up the guiding center orbit calculation (if selected), some of the terms in the ODE are precalculated and tabulated in a file. This is done using code from the https://github.com/dlg0/orbit_tracer repo. After cloning that repo, run the gc_terms.pro
routine on an ar2Input.nc
file, e.g.,
cd ~/scratch/aorsa2d/colestock-kashuba-reference
idl
IDL>@constants
IDL>gc_terms, _me_amu, -1, ar2='input/ar2Input.nc', /oneD